Database Outages

Tagnet keeps everything in its database, so when the database cannot be reached Tagnet cannot do its work. It is built to ride that out without anyone stepping in: it keeps going, tells people, and picks up where it left off as soon as the database is back. This article collects what happens, what people and readers see, how long an outage your readers can absorb, and how to find out afterwards what happened.

It applies to SQL Server installations, where the database lives on its own server and can be patched, restarted or cut off by the network independently of Tagnet. On SQLite the database is a file on the Tagnet server, and the outage to watch for is a full disk, which raises Storage running low (see Alerts).

What Tagnet does during an outage

When	What happens
First seconds	Requests that need the database fail. Readers and API clients get 503 Service Unavailable with Retry-After, never a login or validation error. Tagnet checks the database every 30 seconds.
Within a minute	Once two checks in a row have failed, web pages show Tagnet can't reach its database, refreshing every 30 seconds, and /health/ready reports unavailable.
1 minute	The service log records when the outage began. From here on, the errors that only repeat that the database is unavailable are left out of the log, so a long outage does not fill it.
5 minutes	Tagnet emails everyone its Critical alerts reach: [Critical] Tagnet on <server> cannot reach its database. It sends this directly, because its alerts are kept in the database. See When the database is unavailable.
Database back	Tagnet notices within 30 seconds, clears its connection pool so everything recovers at once, and raises the Database was unreachable alert with the start, end and last error. The log line recording the end says how many repeated errors were left out.

Nothing needs restarting. The Tagnet service stays running throughout.

What people and systems see

• People in the web app: the "can't reach its database" page, which comes back to Tagnet by itself once the database is available. A page that was already open may show an error until it is reloaded.
• Readers: 503 with Retry-After. Readers that buffer hold their reads and send them when Tagnet accepts them again; see How long an outage your readers can absorb.
• Integrations and API clients: 503 with Retry-After on every call that needs the database. Retry after the interval given.
• Outbound integrations and workflows: paused. Deliveries already queued are sent once the database returns.

When Tagnet starts during an outage

If the Tagnet server restarts while the database is down (a patch night where both restart, for example), Tagnet waits for the database however long it takes and starts the moment it answers. While it waits the service shows as Running but the web pages and reader endpoints do not answer yet, so readers see connection errors rather than 503 and buffer the same way. The five-minute email still goes out. See Startup type and recovery for the log lines, and Upgrading from an existing installation for what an upgrade does when the database is down.

One exception: when the Tagnet service runs as a gMSA, Windows cannot start the service at all after a server restart while no domain controller answers. See If a domain controller is not available at startup.

How long an outage your readers can absorb

Readers buffer a fixed number of events, not a length of time, so how long an outage they absorb depends on how busy they are. Roughly one event per tag read:

Reader	Buffer (default)	When it fills
Impinj R700	Event Buffer Size 100,000	The oldest events are dropped and replaced with an overflow marker. See Impinj R700 Setup.
Zebra FX series	Max Events 150,000	Retention stops at the limit. Retained events are also lost if the reader reboots. See Tag Data Retention.

To estimate, divide the buffer by the reader's read rate during the hours an outage could happen:

Average reads per second	R700 (100,000)	Zebra FX (150,000)
1 (a quiet doorway)	about 28 hours	about 42 hours
10 (a busy dock door)	about 2.8 hours	about 4 hours
50 (a conveyor or tunnel)	about 33 minutes	about 50 minutes

If your database maintenance windows run longer than your busiest readers can hold, schedule them outside the busy hours, or reduce what the busiest readers report using their own filtering settings.

Watching Tagnet from outside

If you monitor services with an external tool (a load balancer health probe, SCOM, Uptime Kuma, Nagios and the like), point it at these two addresses. Both answer without signing in and never touch the database:

Address	Answers
/health/live	200 {"status":"live"} while the Tagnet process runs; 503 {"status":"stopping"} while it shuts down.
/health/ready	200 {"status":"ready"} when Tagnet can serve; 503 {"status":"unavailable","reason":"The database is unavailable.","since":"..."} during an outage, with Retry-After.

Use /health/ready to alert on outages and /health/live to tell "Tagnet is down" from "Tagnet is up but its database is not". Point a load balancer or anything that restarts services at /health/live, not /health/ready: taking Tagnet out of rotation or restarting it does not bring the database back, and readers would then get the load balancer's error instead of Tagnet's 503 with Retry-After. Neither says anything about versions or servers, so they are safe to expose to a monitoring network. While Tagnet is still starting, neither answers (see above), which a monitor sees as the service being down. /health/ready watches the SQL Server connection; on a SQLite installation it always reports ready, so watch free disk space there instead.

Finding out what happened afterwards

• The alert: the Alerts inbox keeps Database was unreachable with the start, end, duration and last error, resolved once it was sent.
• The emails: the five-minute email and the alert email, in the inboxes of the people Critical alerts reach.
• The service log (C:\ProgramData\Stratum Global\Tagnet\logs, kept 30 days): search for has not been able to reach database (the start), is reachable again after (the end), and, for an outage at startup, cannot be reached or exists but is not yet available.
• Windows: the System event log shows whether the Tagnet service itself stopped or restarted (events 7034, 7036), which tells a Tagnet failure apart from a database outage.

Related

• Services: startup type, recovery, and the wait at startup
• Alerts: the outage email and the recovery alert
• Service Account: which account keeps Tagnet recovering on its own
• SQL Server Setup: connection errors and what they mean