Bulk importing devices extremely slow

[martijnremmen]

Deployment Type

Self-hosted

NetBox Version

v4.2.6

Python Version

3.12

Steps to Reproduce

1. create all required models for a device (role, manufacturer, status, site)
2. import device_type Catalyst 9200L-48P-4X
3. import 10 devices using this device_type as csv.

Notes:

• I am using a locally hosted Postgres instance in Docker with image postgres:15-alpine

Expected Behavior

the import completing within seconds

Observed Behavior

the import completes in more than 1 minute. Performs more than 11K SQL queries. And gives no progress indicator. for just 10 devices

[jnovinger]

Hey @martijnremmen , thanks for the bug report. While I'm able to reproduce some slowness using your STR (or at least how I interpreted them), I'm not able to recreate the extreme slowness and SQL query count that your screenshot shows.

On my local, with the development server and Postgres (v14.17 via Homebrew) running side-by-side, this took me ~25 seconds and only 20 queries.

This is the CSV I used to do the import of the 10 devices:

name,status,tenant,site,location,rack,role,manufacturer,device_type
Cisco Catalyst 9200L-48P-4X-1,active,,Jason's House,Jason's Desk,,Access Switch,Cisco,Catalyst 9200L-48P-4X
Cisco Catalyst 9200L-48P-4X-2,active,,Jason's House,Jason's Desk,,Access Switch,Cisco,Catalyst 9200L-48P-4X
Cisco Catalyst 9200L-48P-4X-3,active,,Jason's House,Jason's Desk,,Access Switch,Cisco,Catalyst 9200L-48P-4X
Cisco Catalyst 9200L-48P-4X-4,active,,Jason's House,Jason's Desk,,Access Switch,Cisco,Catalyst 9200L-48P-4X
Cisco Catalyst 9200L-48P-4X-5,active,,Jason's House,Jason's Desk,,Access Switch,Cisco,Catalyst 9200L-48P-4X
Cisco Catalyst 9200L-48P-4X-6,active,,Jason's House,Jason's Desk,,Access Switch,Cisco,Catalyst 9200L-48P-4X
Cisco Catalyst 9200L-48P-4X-7,active,,Jason's House,Jason's Desk,,Access Switch,Cisco,Catalyst 9200L-48P-4X
Cisco Catalyst 9200L-48P-4X-8,active,,Jason's House,Jason's Desk,,Access Switch,Cisco,Catalyst 9200L-48P-4X
Cisco Catalyst 9200L-48P-4X-9,active,,Jason's House,Jason's Desk,,Access Switch,Cisco,Catalyst 9200L-48P-4X
Cisco Catalyst 9200L-48P-4X-10,active,,Jason's House,Jason's Desk,,Access Switch,Cisco,Catalyst 9200L-48P-4X

Can you provide any more details that would help us reproduce this?

[kkthxbye-code]

I'm not able to recreate the extreme slowness and SQL query count that your screenshot shows.

On my local, with the development server and Postgres (v14.17 via Homebrew) running side-by-side, this took me ~25 seconds and only 20 queries.

You are inspecting the redirect. You have to go to history in the debug toolbar and switch to the post request.

[jnovinger]

You are inspecting the redirect. You have to go to history in the debug toolbar and switch to the post request.

Doh, thanks, you are correct. I'm seeing similar output now.

[jnovinger]

FWIW, batch delete on the 10 created devices has the same issue--this time with ~16,000 queries.

[martijnremmen]

Thanks for investigating the issue!

also note when removing the interfaces from the device type, importing gets a lot faster with 737 queries (still alot)