Netbox Upgrade Analysis

Record of the NetBox upgrade carried out under sc-10975: the path taken, the breaking changes that applied, and the failure encountered along the way.

Scope

Our NetBox usage is minimal and IPAM-only: we track allocated IP networks and subnets. It is the source of truth referenced by convention comments throughout the KCC VPC manifests, for example k8s/infra-services/tb-infra-mgmt-vpc-project/compute/europe-west1-network.yaml.

That scope is what makes this a straight migration rather than a project. We are not using circuits, device modelling, virtualisation, L2VPN, config contexts, custom scripts, reports, or GraphQL — so the overwhelming majority of the breaking changes across nine releases do not apply to us.

Stepping through the intermediate NetBox version numbers on the path is accepted.

Starting state, before the upgrade

Recorded as it was at the outset. The deployment now runs v4.6.7, with an rqworker, probes, strategy: Recreate, ArgoCD sync-wave ordering and a Redis NetworkPolicy — see Netbox for the current shape.

Database

Cloud SQL, declared via Config Connector in k8s/infra-services/tb-infra-mgmt-project/sql/cloudsql-postgres.yaml:

The netbox database and netbox user are declared in sql/db-list.yaml and sql/db-user.yaml against the same instance.

Prerequisites: already satisfied

This is the important discovery result. PostgreSQL 15 satisfies every version floor on the path, including the future v4.7 requirement. There is no database workstream.

PG floor by release, for reference

Upgrade path

The upstream upgrade guide states that NetBox can be upgraded directly to any newer release except when incrementing a major version, which may only be done from the most recent minor release of the prior major. This accommodates the consolidation of schema migrations at major version boundaries.

v3.5.9 therefore cannot go straight to v4.6.7. That is the hard upstream constraint, but it is not the only one: a direct v3.7.8 → v4.6.7 jump was attempted and failed in production, because the deployed runtime code ran ahead of the schema it was migrating. The path below therefore steps a single minor version at a time.

graph LR
    A["v3.5.9"] --> B["v3.7.8"]
    B --> C["v4.3.7"]
    C --> D["v4.4.10"]
    D --> E["v4.5.10"]
    E --> F["v4.6.7"]

What went wrong going straight to v4.6.7

Attempting v3.7.8 → v4.6.7 failed at users.0005_alter_user_table with a misleading error:

django.db.utils.InternalError: current transaction is aborted,
commands ignored until end of transaction block

That is a consequence, not the cause. The real sequence:

1. users.0005 (a 4.0 migration) calls netboxuser_ct.delete().
2. That fires 4.6’s pre_delete signal → notify_object_changed → ObjectType.get_for_model.
3. 4.6’s config initialisation queries core_configrevision.active.
4. That column is added by core.0019_configrevision_active, introduced in 4.4. At this point the schema is still at core.0010, so the query fails with UndefinedColumn.
5. The failure aborts the migration’s transaction. NetBox catches the exception and logs Skipping config initialization (database unavailable), but does not roll back to a savepoint, so the transaction stays poisoned.
6. The next query in that transaction raises InFailedSqlTransaction, killing the migration.

All eight DDL statements in that migration are fine — they were verified by replaying them inside a rolled-back transaction. The DDL is not the problem, and the reported error names neither the real query nor the real column.

Why v4.3.7 specifically

Two independent constraints intersect at 4.3:

NetBox rejects defining both forms, so the configuration cannot be made to span the range:

elif hasattr(configuration, 'DATABASE') and hasattr(configuration, 'DATABASES'):
    raise ImproperlyConfigured("DATABASE and DATABASES may not be set together.")

Targeting 4.0–4.2 would therefore also require reverting configuration.py to the singular form. v4.3.7 is the only version satisfying both constraints, making it reachable with an image-tag change alone.

Pre-flighting each hop

The underlying fault is not fixed upstream. In 4.3, 4.4 and 4.6 alike, config initialisation still catches DatabaseError and returns without rolling back to a savepoint:

except DatabaseError:
    # The database may not be available yet (e.g. when running a management command)
    logger.warning("Skipping config initialization (database unavailable)")
    return

So the hazard is structural, and single-minor stepping is the mitigation rather than a cure. Before each hop, check whether that trap can be sprung:

1. List the migrations the hop will apply — diff the migrations/ directories between the two tags.
2. Find any RunPython operations in that set. Pure schema operations (CreateModel, AlterField, AddField) cannot trigger it.
3. Check whether those functions fire model signals. Per-object .save() and .delete() do; bulk_update, bulk_create and queryset .update() do not. The v4.6.7 failure was caused by a .delete() firing pre_delete.
4. Check the affected tables are non-empty. A RunPython that iterates zero rows cannot fire anything regardless.

Worked example — the v4.3.7 → v4.4.10 hop, 17 migrations:

Nothing in that set fires the signal path that broke the v4.6.7 attempt, so the hop is low risk.

Every hop runs the same procedure:

1. Take an on-demand Cloud SQL backup of the infra-management instance.
2. Bump the image tag in both netbox-deployment.yaml and netbox-worker.yaml, and merge.
3. The container entrypoint applies migrations on startup.
4. Verify prefixes and aggregates render correctly and the API responds.

Migrations on an IPAM-only dataset of this size complete in seconds, not hours. The on-demand backup is the rollback mechanism — reverting the image tag alone does not undo a migration.

Breaking changes that apply to us

Filtered to our IPAM-only usage. Full upstream release notes are linked at the bottom.

Must action

Be aware, no action needed

Not applicable to us

Listed so the assessment is auditable rather than hand-waved:

• v3.6 Device.device_role → role; napalm fields removed; virtual chassis interface filters
• v3.7 Webhook fields → EventRules; L2VPN moved ipam → vpn app; ConfigRevision moved to core
• v4.0 Legacy reports dropped and converted to custom scripts; GraphQL filter format changed
• v4.1 VirtualMachine.disk / VirtualDisk.size units GB → MB; VLANGroup.min_vid/max_vid → vid_ranges
• v4.2 CircuitTermination, Cluster, CircuitGroupAssignment generic FK changes
• v4.3 Service.parent generic relationship; Contact.group → groups M2M; staged changes removed
• v4.5 cable-terminations API read-only; render_config permission; platform config context inheritance
• All GraphQL syntax changes across v4.0, v4.3 and v4.5 — we do not use GraphQL

Container image changes

Manifest changes

1. DATABASE → DATABASES (required at hop 2)

In base/configmap.yaml:

DATABASES = {
    'default': {
        'NAME': 'netbox',
        'USER': 'netbox',
        'PASSWORD': os.environ.get('DB_PASSWORD'),
        'HOST': '10.17.8.5',
        'PORT': '5432',
        'OPTIONS': {'sslmode': 'require'},
    }
}

Keep the singular form for hop 1; introduce the plural form alongside the v4.6.7 image bump.

2. Add an rqworker Deployment (required at hop 2)

From v4.4, bulk import/modify/delete run as background jobs and housekeeping is a system job. Without a worker these silently never complete. This matters directly for sc-10967, which does bulk subnet reconciliation.

The worker runs the same image and tag as the web pod, with the command overridden:

command:
  - /opt/netbox/venv/bin/python
  - /opt/netbox/netbox/manage.py
  - rqworker

The commented-out block already in base/netbox-deployment.yaml is the starting point.

3. strategy: Recreate (recommended, one line)

With the default RollingUpdate, a new pod can begin migrating the schema while the old pod is still serving through code that expects the old schema. Recreate costs a few seconds of downtime on a single-replica internal tool and removes the overlap entirely.

spec:
  strategy:
    type: Recreate

4. Probes (recommended)

There are currently no probes, so Kubernetes marks the pod ready while the entrypoint is still migrating and nothing is listening on :8080. Add a startupProbe generous enough to cover migration time and a real readinessProbe. Add a livenessProbe only once startup timing is understood, so it cannot kill a slow initialisation.

If ALLOWED_HOSTS is tightened from ['*'], give the probes an accepted Host header rather than reopening the wildcard.

5. Optional tidy-ups

Not blockers, but worth folding in while the manifests are open:

• ALLOWED_HOSTS = ['*'] → ['netbox.infra.titanbay.com'].
• SKIP_SUPERUSER="false" on the long-running Deployment → verify an admin exists, set "true", and drop the SUPERUSER_* env vars.
• SUPERUSER_PASSWORD reuses the same secret key as SECRET_KEY (secret-key in netbox-credentials). Separate and rotate them independently.
• API_TOKEN_PEPPERS — not required while v1 tokens work, but required before v5.0. Must be independent of SECRET_KEY, at least 50 characters, sourced from Secret Manager.
• securityContext — runAsNonRoot, dropped capabilities. Determine the image’s actual netbox UID/GID first; it changed from unit in netbox-docker 4.0.0.

Not needed at our scale

Recorded so these are consciously declined rather than overlooked:

• Splitting Redis into separate task and cache instances. Correct at scale, unnecessary here.
• PersistentVolumes for media/, scripts/, reports/. IPAM-only usage means there should be nothing in them — confirm once during pre-flight, then accept the ephemerality.
• A production-clone rehearsal. Justified for a large dataset; disproportionate for this one. The on-demand backup covers it.
• Gated ArgoCD sync waves with a dedicated migration Job. The entrypoint migration on a small dataset is fine given a backup taken immediately beforehand.

Pre-flight checks

These were established once and hold for the whole campaign.

Check 1 is the significant one: an incompatible plugin can stop Django starting before migrations run.

Check 2 removes an entire risk category. Every v4 API-facing breaking change — removed filters, obsolete nested serializers, the GraphQL syntax changes, and the /api/vpn/ L2VPN move — only affects API callers. It also makes API_TOKEN_PEPPERS optional for now, since tokens are only used by people in the UI.

Per-hop, the migration-plan assessment described under Pre-flighting each hop must also be repeated.

Backup checkpoints

Take a fresh on-demand backup before every hop. A backup only restores the schema as it was when taken, so its value decays with each hop: the original pre-upgrade backup now restores a v3.7.8 schema, which is over a hundred migrations behind.

This is not theoretical. The failed v4.6.7 attempt committed six migrations before dying, which left the database unable to run the previous image — the backup was the only route back.

Progress

Final state verified: 791 migrations applied, migrate --check clean, /login/ returning 200, and 5 Aggregates / 14 Prefixes / 3 Sites intact with all 14 prefixes still scoped to dcim.site. The v4.6 feature tables (dcim_rackgroup, dcim_cablebundle, virtualization_virtualmachinetype) are present.

Six deployments were needed rather than the two originally scoped. The extra four are a direct consequence of the analysis error described above.

Known gaps after v4.6.7

These are deliberately out of scope for the upgrade itself and should be tracked separately:

Probes were deliberately not bundled into hop 1, to keep a not-ready pod attributable to the upgrade rather than the probe configuration. In hindsight that was the wrong trade: it cost detection on the PGSSLCERT failure, where the pod reported 1/1 Running throughout a total outage.

References

• NetBox upgrade guide
• NetBox release notes
• NetBox required configuration parameters
• netbox-docker releases
• Netbox managed service documentation