Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

synapse_port_db fails with "argument of type 'int' is not iterable" #11918

Closed
Xiretza opened this issue Feb 5, 2022 · 3 comments · Fixed by #12037
Closed

synapse_port_db fails with "argument of type 'int' is not iterable" #11918

Xiretza opened this issue Feb 5, 2022 · 3 comments · Fixed by #12037
Assignees
@squahtx
Labels
X-Needs-Info This issue is blocked awaiting information from the reporter

Comments

@Xiretza
Copy link

Xiretza commented Feb 5, 2022

Description

When running synapse_port_db to convert an existing SQLite installation to PostgreSQL, the script fails after a short time with the following output:

synapse_port_db output 
2022-02-05 12:55:44,665 - synapse.config.key - 147 - WARNING - This server is configured to use 'matrix.org' as its trusted key server via the
'trusted_key_servers' config option. 'matrix.org' is a good choice for a key
server since it is long-lived, stable and trusted. However, some admins may
wish to use another server for this purpose.

To suppress this warning and continue using 'matrix.org', admins should set
'suppress_key_server_warning' to 'true' in homeserver.yaml.
--------------------------------------------------------------------------------
2022-02-05 12:55:44,834 - synapse.storage.prepare_database - 116 - INFO - ('main', 'state'): Checking existing schema version
2022-02-05 12:55:44,881 - synapse.storage.prepare_database - 120 - INFO - ('main', 'state'): Existing schema is 67 (+0 deltas)
2022-02-05 12:55:44,881 - synapse.storage.databases.main - 355 - INFO - Checking database for consistency with configuration...
2022-02-05 12:55:44,882 - synapse.storage.prepare_database - 412 - INFO - Applying schema deltas for v67
2022-02-05 12:55:44,884 - synapse.storage.prepare_database - 544 - INFO - Schema now up to date
2022-02-05 12:55:44,954 - synapse.storage.util.id_generators - 88 - INFO - Initialising stream generator for pushers(id): 1
2022-02-05 12:55:44,956 - synapse.storage.util.id_generators - 88 - INFO - Initialising stream generator for deleted_pushers(stream_id): 1
2022-02-05 12:55:44,962 - synapse.storage.util.id_generators - 88 - INFO - Initialising stream generator for events(stream_ordering): 2772100
2022-02-05 12:55:44,964 - synapse.storage.util.id_generators - 88 - INFO - Initialising stream generator for events(stream_ordering): -66907
2022-02-05 12:55:44,967 - synapse.storage.util.id_generators - 88 - INFO - Initialising stream generator for ex_outlier_stream(event_stream_ordering): -66908
2022-02-05 12:55:44,971 - synapse.metrics - 678 - INFO - Collecting gc 1
2022-02-05 12:55:45,093 - synapse.storage.prepare_database - 116 - INFO - ('main', 'state'): Checking existing schema version
2022-02-05 12:55:45,104 - synapse.storage.prepare_database - 120 - INFO - ('main', 'state'): Existing schema is 67 (+0 deltas)
2022-02-05 12:55:45,104 - synapse.storage.databases.main - 355 - INFO - Checking database for consistency with configuration...
2022-02-05 12:55:45,110 - synapse.storage.prepare_database - 412 - INFO - Applying schema deltas for v67
2022-02-05 12:55:45,112 - synapse.storage.prepare_database - 544 - INFO - Schema now up to date
2022-02-05 12:55:45,126 - synapse.storage.util.id_generators - 88 - INFO - Initialising stream generator for pushers(id): 1
2022-02-05 12:55:45,127 - synapse.storage.util.id_generators - 88 - INFO - Initialising stream generator for deleted_pushers(stream_id): 1
2022-02-05 12:55:45,359 - synapse_port_db - 664 - INFO - Found 152 tables
2022-02-05 13:04:41,177 - synapse_port_db - 332 - INFO - Table user_directory: 0/33573 (rows 1-0) already ported
2022-02-05 13:04:41,179 - synapse_port_db - 332 - INFO - Table pushers: 0/0 (rows 1-0) already ported
2022-02-05 13:04:41,185 - synapse_port_db - 332 - INFO - Table stream_positions: 0/0 (rows 1-0) already ported
2022-02-05 13:04:41,185 - synapse_port_db - 332 - INFO - Table local_group_updates: 0/2 (rows 1-0) already ported
2022-02-05 13:04:41,186 - synapse_port_db - 332 - INFO - Table room_account_data: 0/135 (rows 1-0) already ported
2022-02-05 13:04:41,188 - synapse_port_db - 332 - INFO - Table event_push_summary_stream_ordering: 0/1 (rows 1-0) already ported
2022-02-05 13:04:41,189 - synapse_port_db - 332 - INFO - Table monthly_active_users: 0/0 (rows 1-0) already ported
2022-02-05 13:04:41,190 - synapse_port_db - 332 - INFO - Table room_depth: 0/154 (rows 1-0) already ported
2022-02-05 13:04:41,191 - synapse_port_db - 332 - INFO - Table users_in_public_rooms: 0/41067 (rows 1-0) already ported
2022-02-05 13:04:41,194 - synapse_port_db - 332 - INFO - Table group_rooms: 0/0 (rows 1-0) already ported
2022-02-05 13:04:41,194 - synapse_port_db - 332 - INFO - Table remote_media_cache: 11588/12419 (rows 1-11588) already ported
2022-02-05 13:04:41,195 - synapse_port_db - 332 - INFO - Table event_to_state_groups: 444000/963245 (rows 1-446838) already ported
2022-02-05 13:04:41,197 - synapse_port_db - 332 - INFO - Table applied_module_schemas: 0/0 (rows 1-0) already ported
2022-02-05 13:04:41,197 - synapse_port_db - 332 - INFO - Table presence_stream: 4168/10840 (rows 1-22312624) already ported
2022-02-05 13:04:41,198 - synapse_port_db - 332 - INFO - Table group_attestations_renewals: 0/0 (rows 1-0) already ported
2022-02-05 13:04:41,204 - synapse_port_db - 332 - INFO - Table state_events: 245376/276591 (rows 1-264677) already ported
2022-02-05 13:04:41,204 - synapse_port_db - 332 - INFO - Table ratelimit_override: 0/0 (rows 1-0) already ported
2022-02-05 13:04:41,205 - synapse_port_db - 332 - INFO - Table batch_events: 0/0 (rows 1-0) already ported
2022-02-05 13:04:41,205 - synapse_port_db - 332 - INFO - Table room_stats_state: 0/154 (rows 1-0) already ported
2022-02-05 13:04:41,206 - synapse_port_db - 332 - INFO - Table device_federation_inbox: 0/1525 (rows 1-0) already ported
2022-02-05 13:04:41,207 - synapse_port_db - 332 - INFO - Table state_groups: 101281/126611 (rows 1-102652) already ported
2022-02-05 13:04:41,210 - synapse_port_db - 332 - INFO - Table room_memberships: 241453/271194 (rows 1-260618) already ported
2022-02-05 13:04:41,211 - synapse_port_db - 332 - INFO - Table receipts_graph: 0/28770 (rows 1-0) already ported
2022-02-05 13:04:41,213 - synapse_port_db - 332 - INFO - Table device_federation_outbox: 0/0 (rows 1-0) already ported
2022-02-05 13:04:41,214 - synapse_port_db - 332 - INFO - Table local_media_repository_url_cache: 0/0 (rows 1-0) already ported
2022-02-05 13:04:41,214 - synapse_port_db - 332 - INFO - Table room_tags: 0/2 (rows 1-0) already ported
2022-02-05 13:04:41,216 - synapse_port_db - 332 - INFO - Table group_users: 0/0 (rows 1-0) already ported
2022-02-05 13:04:41,217 - synapse_port_db - 332 - INFO - Table event_auth_chain_to_calculate: 0/40 (rows 1-0) already ported
2022-02-05 13:04:41,217 - synapse_port_db - 332 - INFO - Table application_services_state: 0/1 (rows 1-0) already ported
2022-02-05 13:04:41,218 - synapse_port_db - 332 - INFO - Table redactions: 26502/27881 (rows 1-26505) already ported
2022-02-05 13:04:41,219 - synapse_port_db - 332 - INFO - Table event_push_actions: 0/227 (rows 1-0) already ported
2022-02-05 13:04:41,220 - synapse_port_db - 332 - INFO - Table presence: 0/0 (rows 1-0) already ported
2022-02-05 13:04:41,221 - synapse_port_db - 332 - INFO - Table insertion_event_edges: 0/0 (rows 1-0) already ported
2022-02-05 13:04:41,222 - synapse_port_db - 332 - INFO - Table insertion_events: 0/0 (rows 1-0) already ported
2022-02-05 13:04:41,224 - synapse_port_db - 332 - INFO - Table current_state_events: 0/69575 (rows 1-0) already ported
2022-02-05 13:04:41,225 - synapse_port_db - 332 - INFO - Table group_summary_rooms: 0/0 (rows 1-0) already ported
2022-02-05 13:04:41,226 - synapse_port_db - 332 - INFO - Table e2e_room_keys: 0/1336 (rows 1-0) already ported
2022-02-05 13:04:41,227 - synapse_port_db - 332 - INFO - Table remote_profile_cache: 0/0 (rows 1-0) already ported
2022-02-05 13:04:41,228 - synapse_port_db - 332 - INFO - Table device_lists_outbound_last_success: 0/2962 (rows 1-0) already ported
2022-02-05 13:04:41,230 - synapse_port_db - 332 - INFO - Table receipts_linearized: 0/28771 (rows 1-0) already ported
2022-02-05 13:04:41,232 - synapse_port_db - 332 - INFO - Table events: 387000/1086444 (rows -64760-437060) already ported
2022-02-05 13:04:41,233 - synapse_port_db - 332 - INFO - Table remote_media_cache_thumbnails: 46488/54681 (rows 1-46488) already ported
2022-02-05 13:04:41,235 - synapse_port_db - 332 - INFO - Table blocked_rooms: 0/0 (rows 1-0) already ported
2022-02-05 13:04:41,236 - synapse_port_db - 332 - INFO - Table event_relations: 0/51357 (rows 1-0) already ported
2022-02-05 13:04:41,237 - synapse_port_db - 332 - INFO - Table user_filters: 0/5 (rows 1-0) already ported
2022-02-05 13:04:41,237 - synapse_port_db - 332 - INFO - Table threepid_validation_session: 0/1 (rows 1-0) already ported
2022-02-05 13:04:41,239 - synapse_port_db - 332 - INFO - Table dehydrated_devices: 0/0 (rows 1-0) already ported
2022-02-05 13:04:41,240 - synapse_port_db - 332 - INFO - Table e2e_one_time_keys_json: 0/330 (rows 1-0) already ported
2022-02-05 13:04:41,242 - synapse_port_db - 332 - INFO - Table users: 16/16 (rows 1-16) already ported
2022-02-05 13:04:41,243 - synapse_port_db - 332 - INFO - Table stream_ordering_to_exterm: 118642/277197 (rows 1-1705616) already ported
2022-02-05 13:04:41,244 - synapse_port_db - 332 - INFO - Table push_rules: 0/13 (rows 1-0) already ported
2022-02-05 13:04:41,246 - synapse_port_db - 332 - INFO - Table device_lists_remote_resync: 0/31 (rows 1-0) already ported
2022-02-05 13:04:41,247 - synapse_port_db - 332 - INFO - Table event_push_summary: 0/1 (rows 1-0) already ported
2022-02-05 13:04:41,249 - synapse_port_db - 332 - INFO - Table user_daily_visits: 0/984 (rows 1-0) already ported
2022-02-05 13:04:41,249 - synapse_port_db - 332 - INFO - Table event_edges: 430000/1209485 (rows 1-450860) already ported
2022-02-05 13:04:41,249 - synapse_port_db - 332 - INFO - Table device_inbox: 0/4339 (rows 1-0) already ported
2022-02-05 13:04:41,250 - synapse_port_db - 332 - INFO - Table device_lists_remote_cache: 0/73405 (rows 1-0) already ported
2022-02-05 13:04:41,250 - synapse_port_db - 332 - INFO - Table rooms: 132/159 (rows 1-144) already ported
2022-02-05 13:04:41,251 - synapse_port_db - 332 - INFO - Table erased_users: 0/0 (rows 1-0) already ported
2022-02-05 13:04:41,251 - synapse_port_db - 332 - INFO - Table local_current_membership: 0/184 (rows 1-0) already ported
2022-02-05 13:04:41,252 - synapse_port_db - 332 - INFO - Table insertion_event_extremities: 0/0 (rows 1-0) already ported
2022-02-05 13:04:41,252 - synapse_port_db - 332 - INFO - Table room_aliases: 0/6 (rows 1-0) already ported
2022-02-05 13:04:41,253 - synapse_port_db - 332 - INFO - Table destinations: 0/4483 (rows 1-0) already ported
2022-02-05 13:04:41,253 - synapse_port_db - 332 - INFO - Table room_tags_revisions: 0/3 (rows 1-0) already ported
2022-02-05 13:04:41,254 - synapse_port_db - 332 - INFO - Table user_signature_stream: 0/7 (rows 1-0) already ported
2022-02-05 13:04:41,255 - synapse_port_db - 332 - INFO - Table users_who_share_private_rooms: 0/176 (rows 1-0) already ported
2022-02-05 13:04:41,255 - synapse_port_db - 332 - INFO - Table e2e_fallback_keys_json: 0/5 (rows 1-0) already ported
2022-02-05 13:04:41,256 - synapse_port_db - 332 - INFO - Table event_reports: 0/0 (rows 1-0) already ported
2022-02-05 13:04:41,256 - synapse_port_db - 332 - INFO - Table public_room_list_stream: 3/3 (rows 1-3) already ported
2022-02-05 13:04:41,256 - synapse_port_db - 332 - INFO - Table group_summary_roles: 0/0 (rows 1-0) already ported
2022-02-05 13:04:41,257 - synapse_port_db - 332 - INFO - Table room_alias_servers: 0/7 (rows 1-0) already ported
2022-02-05 13:04:41,257 - synapse_port_db - 332 - INFO - Table device_lists_remote_extremeties: 0/13308 (rows 1-0) already ported
2022-02-05 13:04:41,258 - synapse_port_db - 332 - INFO - Table ignored_users: 0/0 (rows 1-0) already ported
2022-02-05 13:04:41,258 - synapse_port_db - 332 - INFO - Table group_invites: 0/0 (rows 1-0) already ported
2022-02-05 13:04:41,258 - synapse_port_db - 332 - INFO - Table state_group_edges: 96665/120820 (rows 1-98013) already ported
2022-02-05 13:04:41,258 - synapse_port_db - 332 - INFO - Table e2e_cross_signing_signatures: 0/18 (rows 1-0) already ported
2022-02-05 13:04:41,259 - synapse_port_db - 332 - INFO - Table group_summary_room_categories: 0/0 (rows 1-0) already ported
2022-02-05 13:04:41,259 - synapse_port_db - 332 - INFO - Table threepid_validation_token: 0/0 (rows 1-0) already ported
2022-02-05 13:04:41,260 - synapse_port_db - 332 - INFO - Table e2e_room_keys_versions: 0/1 (rows 1-0) already ported
2022-02-05 13:04:41,260 - synapse_port_db - 332 - INFO - Table application_services_txns: 0/0 (rows 1-0) already ported
2022-02-05 13:04:41,261 - synapse_port_db - 332 - INFO - Table sessions: 0/0 (rows 1-0) already ported
2022-02-05 13:04:41,261 - synapse_port_db - 332 - INFO - Table local_media_repository: 210/216 (rows 1-210) already ported
2022-02-05 13:04:41,262 - synapse_port_db - 332 - INFO - Table background_updates: 0/0 (rows 1-0) already ported
2022-02-05 13:04:41,262 - synapse_port_db - 332 - INFO - Table event_backward_extremities: 0/2464 (rows 1-0) already ported
2022-02-05 13:04:41,263 - synapse_port_db - 332 - INFO - Table ui_auth_sessions: 0/0 (rows 1-0) already ported
2022-02-05 13:04:41,263 - synapse_port_db - 332 - INFO - Table e2e_device_keys_json: 0/7 (rows 1-0) already ported
2022-02-05 13:04:41,264 - synapse_port_db - 332 - INFO - Table refresh_tokens: 0/0 (rows 1-0) already ported
2022-02-05 13:04:41,264 - synapse_port_db - 332 - INFO - Table push_rules_enable: 0/13 (rows 1-0) already ported
2022-02-05 13:04:41,264 - synapse_port_db - 332 - INFO - Table room_retention: 0/0 (rows 1-0) already ported
2022-02-05 13:04:41,265 - synapse_port_db - 332 - INFO - Table device_lists_stream: 0/146908 (rows 1-0) already ported
2022-02-05 13:04:41,266 - synapse_port_db - 332 - INFO - Table threepid_guest_access_tokens: 0/0 (rows 1-0) already ported
2022-02-05 13:04:41,266 - synapse_port_db - 332 - INFO - Table server_keys_json: 0/8770 (rows 1-0) already ported
2022-02-05 13:04:41,266 - synapse_port_db - 332 - INFO - Table appservice_room_list: 0/0 (rows 1-0) already ported
2022-02-05 13:04:41,268 - synapse_port_db - 332 - INFO - Table user_stats_current: 0/16 (rows 1-0) already ported
2022-02-05 13:04:41,268 - synapse_port_db - 332 - INFO - Table group_roles: 0/0 (rows 1-0) already ported
2022-02-05 13:04:41,269 - synapse_port_db - 332 - INFO - Table account_validity: 0/0 (rows 1-0) already ported
2022-02-05 13:04:41,269 - synapse_port_db - 332 - INFO - Table local_group_membership: 0/1 (rows 1-0) already ported
2022-02-05 13:04:41,269 - synapse_port_db - 332 - INFO - Table event_search: 386000/796913 (rows 1-387322) already ported
2022-02-05 13:04:41,270 - synapse_port_db - 332 - INFO - Table pusher_throttle: 0/0 (rows 1-0) already ported
2022-02-05 13:04:41,270 - synapse_port_db - 332 - INFO - Table device_lists_outbound_pokes: 0/3573 (rows 1-0) already ported
2022-02-05 13:04:41,271 - synapse_port_db - 332 - INFO - Table profiles: 0/16 (rows 1-0) already ported
2022-02-05 13:04:41,271 - synapse_port_db - 332 - INFO - Table federation_stream_position: 0/2 (rows 1-0) already ported
2022-02-05 13:04:41,272 - synapse_port_db - 332 - INFO - Table user_directory_search: 0/33573 (rows 1-0) already ported
2022-02-05 13:04:41,272 - synapse_port_db - 332 - INFO - Table appservice_stream_position: 0/1 (rows 1-0) already ported
2022-02-05 13:04:41,273 - synapse_port_db - 332 - INFO - Table event_forward_extremities: 0/167 (rows 1-0) already ported
2022-02-05 13:04:41,273 - synapse_port_db - 332 - INFO - Table event_auth_chains: 0/276551 (rows 1-0) already ported
2022-02-05 13:04:41,274 - synapse_port_db - 332 - INFO - Table room_stats_earliest_token: 0/0 (rows 1-0) already ported
2022-02-05 13:04:41,274 - synapse_port_db - 332 - INFO - Table registration_tokens: 0/0 (rows 1-0) already ported
2022-02-05 13:04:41,274 - synapse_port_db - 332 - INFO - Table server_signature_keys: 0/7450 (rows 1-0) already ported
2022-02-05 13:04:41,275 - synapse_port_db - 332 - INFO - Table groups: 0/0 (rows 1-0) already ported
2022-02-05 13:04:41,275 - synapse_port_db - 332 - INFO - Table state_groups_state: 428000/19773199 (rows 1-428079) already ported
2022-02-05 13:04:41,276 - synapse_port_db - 332 - INFO - Table account_data: 0/15 (rows 1-0) already ported
2022-02-05 13:04:41,276 - synapse_port_db - 332 - INFO - Table rejections: 112/239 (rows 1-112) already ported
2022-02-05 13:04:41,276 - synapse_port_db - 332 - INFO - Table devices: 0/31 (rows 1-0) already ported
2022-02-05 13:04:41,277 - synapse_port_db - 332 - INFO - Table event_auth_chain_links: 0/561112 (rows 1-0) already ported
2022-02-05 13:04:41,277 - synapse_port_db - 332 - INFO - Table group_attestations_remote: 0/0 (rows 1-0) already ported
2022-02-05 13:04:41,278 - synapse_port_db - 332 - INFO - Table push_rules_stream: 37/37 (rows 1-37) already ported
2022-02-05 13:04:41,278 - synapse_port_db - 332 - INFO - Table received_transactions: 403000/3552865 (rows 1-25873373) already ported
2022-02-05 13:04:41,278 - synapse_port_db - 332 - INFO - Table event_json: 393000/1130759 (rows 1-413785) already ported
2022-02-05 13:04:41,279 - synapse_port_db - 332 - INFO - Table event_reference_hashes: 432000/1130520 (rows 1-452785) already ported
2022-02-05 13:04:41,279 - synapse_port_db - 332 - INFO - Table room_stats_current: 0/154 (rows 1-0) already ported
2022-02-05 13:04:41,280 - synapse_port_db - 332 - INFO - Table current_state_delta_stream: 0/218076 (rows 1-0) already ported
2022-02-05 13:04:41,280 - synapse_port_db - 332 - INFO - Table federation_inbound_events_staging: 0/1 (rows 1-0) already ported
2022-02-05 13:04:41,281 - synapse_port_db - 332 - INFO - Table schema_compat_version: 0/1 (rows 1-0) already ported
2022-02-05 13:04:41,282 - synapse_port_db - 332 - INFO - Table device_auth_providers: 0/0 (rows 1-0) already ported
2022-02-05 13:04:41,282 - synapse_port_db - 332 - INFO - Table local_media_repository_thumbnails: 778/778 (rows 1-778) already ported
2022-02-05 13:04:41,282 - synapse_port_db - 332 - INFO - Table e2e_cross_signing_keys: 0/85905 (rows 1-0) already ported
2022-02-05 13:04:41,283 - synapse_port_db - 332 - INFO - Table event_labels: 0/0 (rows 1-0) already ported
2022-02-05 13:04:41,283 - synapse_port_db - 332 - INFO - Table user_threepid_id_server: 0/0 (rows 1-0) already ported
2022-02-05 13:04:41,284 - synapse_port_db - 332 - INFO - Table stats_incremental_position: 0/1 (rows 1-0) already ported
2022-02-05 13:04:41,284 - synapse_port_db - 332 - INFO - Table worker_locks: 0/0 (rows 1-0) already ported
2022-02-05 13:04:41,284 - synapse_port_db - 332 - INFO - Table user_directory_stream_pos: 0/1 (rows 1-0) already ported
2022-02-05 13:04:41,285 - synapse_port_db - 332 - INFO - Table user_threepids: 0/0 (rows 1-0) already ported
2022-02-05 13:04:41,292 - synapse_port_db - 332 - INFO - Table event_expiry: 0/0 (rows 1-0) already ported
2022-02-05 13:04:41,294 - synapse_port_db - 332 - INFO - Table user_ips: 0/41 (rows 1-0) already ported
2022-02-05 13:04:41,294 - synapse_port_db - 332 - INFO - Table ex_outlier_stream: 11184/12472 (rows 1-11184) already ported
2022-02-05 13:04:41,295 - synapse_port_db - 332 - INFO - Table event_auth: 444000/938618 (rows 1-506665) already ported
2022-02-05 13:04:41,295 - synapse_port_db - 332 - INFO - Table group_summary_users: 0/0 (rows 1-0) already ported
2022-02-05 13:04:41,296 - synapse_port_db - 332 - INFO - Table user_external_ids: 0/0 (rows 1-0) already ported
2022-02-05 13:04:41,296 - synapse_port_db - 332 - INFO - Table deleted_pushers: 0/0 (rows 1-0) already ported
2022-02-05 13:04:41,297 - synapse_port_db - 332 - INFO - Table open_id_tokens: 0/26 (rows 1-0) already ported
2022-02-05 13:04:41,297 - synapse_port_db - 332 - INFO - Table users_pending_deactivation: 0/0 (rows 1-0) already ported
2022-02-05 13:04:41,298 - synapse_port_db - 332 - INFO - Table event_push_actions_staging: 0/10 (rows 1-0) already ported
2022-02-05 13:04:41,298 - synapse_port_db - 332 - INFO - Table group_room_categories: 0/0 (rows 1-0) already ported
2022-02-05 13:04:47,838 - synapse.metrics - 678 - INFO - Collecting gc 1
2022-02-05 13:04:52,809 - synapse_port_db - 722 - ERROR -
Traceback (most recent call last):
  File "/usr/bin/synapse_port_db", line 701, in run
    await make_deferred_yieldable(
twisted.internet.defer.FirstError: FirstError[#97, [Failure instance: Traceback: <class 'TypeError'>: argument of type 'int' is not iterable
/usr/lib/python3.10/site-packages/twisted/internet/defer.py:701:errback
/usr/lib/python3.10/site-packages/twisted/internet/defer.py:764:_startRunCallbacks
/usr/lib/python3.10/site-packages/twisted/internet/defer.py:858:_runCallbacks
/usr/lib/python3.10/site-packages/twisted/internet/defer.py:1751:gotResult
--- <exception caught here> ---
/usr/lib/python3.10/site-packages/twisted/internet/defer.py:1657:_inlineCallbacks
/usr/lib/python3.10/site-packages/twisted/python/failure.py:500:throwExceptionIntoGenerator
/usr/bin/synapse_port_db:347:handle_table
/usr/bin/synapse_port_db:502:handle_search_table
/usr/lib/python3.10/site-packages/synapse/storage/database.py:741:runInteraction
/usr/lib/python3.10/site-packages/synapse/storage/database.py:846:runWithConnection
/usr/lib/python3.10/site-packages/twisted/python/threadpool.py:238:inContext
/usr/lib/python3.10/site-packages/twisted/python/threadpool.py:254:<lambda>
/usr/lib/python3.10/site-packages/twisted/python/context.py:118:callWithContext
/usr/lib/python3.10/site-packages/twisted/python/context.py:83:callWithContext
/usr/lib/python3.10/site-packages/twisted/enterprise/adbapi.py:293:_runWithConnection
/usr/lib/python3.10/site-packages/twisted/python/deprecate.py:298:deprecatedFunction
/usr/lib/python3.10/site-packages/twisted/python/compat.py:404:reraise
/usr/lib/python3.10/site-packages/twisted/enterprise/adbapi.py:284:_runWithConnection
/usr/lib/python3.10/site-packages/synapse/storage/database.py:841:inner_func
/usr/lib/python3.10/site-packages/synapse/storage/database.py:609:new_transaction
/usr/bin/synapse_port_db:471:insert
]]
Traceback (most recent call last):
  File "/usr/bin/synapse_port_db", line 701, in run
    await make_deferred_yieldable(
twisted.internet.defer.FirstError: FirstError[#97, [Failure instance: Traceback: <class 'TypeError'>: argument of type 'int' is not iterable
/usr/lib/python3.10/site-packages/twisted/internet/defer.py:701:errback
/usr/lib/python3.10/site-packages/twisted/internet/defer.py:764:_startRunCallbacks
/usr/lib/python3.10/site-packages/twisted/internet/defer.py:858:_runCallbacks
/usr/lib/python3.10/site-packages/twisted/internet/defer.py:1751:gotResult
--- <exception caught here> ---
/usr/lib/python3.10/site-packages/twisted/internet/defer.py:1657:_inlineCallbacks
/usr/lib/python3.10/site-packages/twisted/python/failure.py:500:throwExceptionIntoGenerator
/usr/bin/synapse_port_db:347:handle_table
/usr/bin/synapse_port_db:502:handle_search_table
/usr/lib/python3.10/site-packages/synapse/storage/database.py:741:runInteraction
/usr/lib/python3.10/site-packages/synapse/storage/database.py:846:runWithConnection
/usr/lib/python3.10/site-packages/twisted/python/threadpool.py:238:inContext
/usr/lib/python3.10/site-packages/twisted/python/threadpool.py:254:<lambda>
/usr/lib/python3.10/site-packages/twisted/python/context.py:118:callWithContext
/usr/lib/python3.10/site-packages/twisted/python/context.py:83:callWithContext
/usr/lib/python3.10/site-packages/twisted/enterprise/adbapi.py:293:_runWithConnection
/usr/lib/python3.10/site-packages/twisted/python/deprecate.py:298:deprecatedFunction
/usr/lib/python3.10/site-packages/twisted/python/compat.py:404:reraise
/usr/lib/python3.10/site-packages/twisted/enterprise/adbapi.py:284:_runWithConnection
/usr/lib/python3.10/site-packages/synapse/storage/database.py:841:inner_func
/usr/lib/python3.10/site-packages/synapse/storage/database.py:609:new_transaction
/usr/bin/synapse_port_db:471:insert
]]
FirstError[#97, [Failure instance: Traceback: <class 'TypeError'>: argument of type 'int' is not iterable
/usr/lib/python3.10/site-packages/twisted/internet/defer.py:701:errback
/usr/lib/python3.10/site-packages/twisted/internet/defer.py:764:_startRunCallbacks
/usr/lib/python3.10/site-packages/twisted/internet/defer.py:858:_runCallbacks
/usr/lib/python3.10/site-packages/twisted/internet/defer.py:1751:gotResult
--- <exception caught here> ---
/usr/lib/python3.10/site-packages/twisted/internet/defer.py:1657:_inlineCallbacks
/usr/lib/python3.10/site-packages/twisted/python/failure.py:500:throwExceptionIntoGenerator
/usr/bin/synapse_port_db:347:handle_table
/usr/bin/synapse_port_db:502:handle_search_table
/usr/lib/python3.10/site-packages/synapse/storage/database.py:741:runInteraction
/usr/lib/python3.10/site-packages/synapse/storage/database.py:846:runWithConnection
/usr/lib/python3.10/site-packages/twisted/python/threadpool.py:238:inContext
/usr/lib/python3.10/site-packages/twisted/python/threadpool.py:254:<lambda>
/usr/lib/python3.10/site-packages/twisted/python/context.py:118:callWithContext
/usr/lib/python3.10/site-packages/twisted/python/context.py:83:callWithContext
/usr/lib/python3.10/site-packages/twisted/enterprise/adbapi.py:293:_runWithConnection
/usr/lib/python3.10/site-packages/twisted/python/deprecate.py:298:deprecatedFunction
/usr/lib/python3.10/site-packages/twisted/python/compat.py:404:reraise
/usr/lib/python3.10/site-packages/twisted/enterprise/adbapi.py:284:_runWithConnection
/usr/lib/python3.10/site-packages/synapse/storage/database.py:841:inner_func
/usr/lib/python3.10/site-packages/synapse/storage/database.py:609:new_transaction
/usr/bin/synapse_port_db:471:insert
]]Preparing sqlite3...
Preparing psycopg2...
Creating port tables...
Setting up sequence generators...
Fetching tables...
Checking on port progress...
Copying to postgres...
user_directory: 100% (33573/33573)
users_who_share_private_rooms: 100% (176/176)
user_directory_search: 100% (33573/33573)
user_directory_stream_pos: 100% (1/1)
event_push_summary_stream_ordering: 100% (1/1)
local_group_updates: 100% (2/2)
room_depth: 100% (154/154)
room_account_data: 100% (135/135)
room_stats_state: 100% (154/154)
users_in_public_rooms: 2% (1000/41067)
event_to_state_groups: 46% (445000/963245)
remote_media_cache: 100% (12419/12419)
room_tags: 100% (2/2)
application_services_state: 100% (1/1)
event_auth_chain_to_calculate: 100% (40/40)
state_events: 89% (246376/276591)
device_federation_inbox: 65% (1000/1525)
presence_stream: 47% (5168/10840)
state_groups: 80% (102281/126611)
event_push_actions: 100% (227/227)
room_memberships: 89% (242453/271194)
receipts_graph: 3% (1000/28770)
redactions: 98% (27502/27881)
current_state_events: 1% (1000/69575)
device_lists_outbound_last_success: 33% (1000/2962)
e2e_room_keys: 74% (1000/1336)
user_filters: 100% (5/5)
receipts_linearized: 3% (1000/28771)
threepid_validation_session: 100% (1/1)
push_rules: 100% (13/13)
device_lists_remote_resync: 100% (31/31)
event_push_summary: 100% (1/1)
e2e_one_time_keys_json: 100% (330/330)
event_relations: 1% (1000/51357)
remote_media_cache_thumbnails: 86% (47488/54681)
stream_ordering_to_exterm: 43% (119642/277197)
user_daily_visits: 100% (984/984)
rooms: 100% (159/159)
event_edges: 35% (431000/1209485)
room_aliases: 100% (6/6)
local_current_membership: 100% (184/184)
room_tags_revisions: 100% (3/3)
user_signature_stream: 100% (7/7)
e2e_fallback_keys_json: 100% (5/5)
room_alias_servers: 100% (7/7)
device_inbox: 23% (1000/4339)
e2e_cross_signing_signatures: 100% (18/18)
e2e_room_keys_versions: 100% (1/1)
events: 35% (389000/1086444)
local_media_repository: 100% (216/216)
push_rules_enable: 100% (13/13)
e2e_device_keys_json: 100% (7/7)
device_lists_remote_cache: 1% (1000/73405)
device_lists_remote_extremeties: 7% (1000/13308)
destinations: 22% (1000/4483)
state_group_edges: 80% (97665/120820)
local_group_membership: 100% (1/1)
user_stats_current: 100% (16/16)
event_backward_extremities: 40% (1000/2464)
device_lists_stream: 0% (1000/146908)
server_keys_json: 11% (1000/8770)

The main error seems to be this:

Traceback (most recent call last):
  File "/usr/bin/synapse_port_db", line 701, in run
    await make_deferred_yieldable(
twisted.internet.defer.FirstError: FirstError[#97, [Failure instance: Traceback: <class 'TypeError'>: argument of type 'int' is not iterable
/usr/lib/python3.10/site-packages/twisted/internet/defer.py:701:errback
/usr/lib/python3.10/site-packages/twisted/internet/defer.py:764:_startRunCallbacks
/usr/lib/python3.10/site-packages/twisted/internet/defer.py:858:_runCallbacks
/usr/lib/python3.10/site-packages/twisted/internet/defer.py:1751:gotResult
--- <exception caught here> ---
/usr/lib/python3.10/site-packages/twisted/internet/defer.py:1657:_inlineCallbacks
/usr/lib/python3.10/site-packages/twisted/python/failure.py:500:throwExceptionIntoGenerator
/usr/bin/synapse_port_db:347:handle_table
/usr/bin/synapse_port_db:502:handle_search_table
/usr/lib/python3.10/site-packages/synapse/storage/database.py:741:runInteraction
/usr/lib/python3.10/site-packages/synapse/storage/database.py:846:runWithConnection
/usr/lib/python3.10/site-packages/twisted/python/threadpool.py:238:inContext
/usr/lib/python3.10/site-packages/twisted/python/threadpool.py:254:<lambda>
/usr/lib/python3.10/site-packages/twisted/python/context.py:118:callWithContext
/usr/lib/python3.10/site-packages/twisted/python/context.py:83:callWithContext
/usr/lib/python3.10/site-packages/twisted/enterprise/adbapi.py:293:_runWithConnection
/usr/lib/python3.10/site-packages/twisted/python/deprecate.py:298:deprecatedFunction
/usr/lib/python3.10/site-packages/twisted/python/compat.py:404:reraise
/usr/lib/python3.10/site-packages/twisted/enterprise/adbapi.py:284:_runWithConnection
/usr/lib/python3.10/site-packages/synapse/storage/database.py:841:inner_func
/usr/lib/python3.10/site-packages/synapse/storage/database.py:609:new_transaction
/usr/bin/synapse_port_db:471:insert
]]

Steps to reproduce

  • systemctl stop synapse.service
  • cp --reflink /var/lib/synapse/homeserver.db{,.snapshot}
  • systemctl start synapse.service
  • synapse_port_db --sqlite-database /var/lib/synapse/homeserver.db.snapshot --postgres-config /etc/synapse/homeserver_postgres.yaml

Version information

  • Homeserver: xiretza.xyz
  • Version: 1.50.2
  • Install method: pacman -S matrix-synapse
  • Platform: Arch Linux
@squahtx squahtx self-assigned this Feb 18, 2022
@squahtx squahtx mentioned this issue Feb 18, 2022
Merged
4 tasks
@squahtx
Copy link
Contributor

squahtx commented Feb 18, 2022

Can you try using the copy of synapse_port_db from https://github.com/matrix-org/synapse/blob/7b9d8066e050400ff0c13e2f553fb428f7539029/scripts/synapse_port_db and let me know if that works?

@squahtx squahtx added the X-Needs-Info This issue is blocked awaiting information from the reporter label Feb 18, 2022
@squahtx
Copy link
Contributor

squahtx commented Feb 24, 2022

Please reopen the issue if the above fix does not work for you!

@Xiretza
Copy link
Author

Xiretza commented Feb 25, 2022

Sorry for the lack of response, I haven't had the time to actually test this until now - I just ran the patched script you linked (not the actual PR though) overnight, and it worked splendidly; I did get two warnings about dropped search rows as expected. I never knew synapse could feel as snappy as it does now! ;)

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@squahtx squahtx

Labels
X-Needs-Info This issue is blocked awaiting information from the reporter
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Handle non-strings in the event_search table in synapse_port_db matrix-org/synapse
2 participants
@Xiretza @squahtx