Start your free Preview
ProductHow it worksSee what's realFix it safelyCatch attacksGraphneIntegrationsPlay the graphWeekly puzzlePricing
IndustriesHealthcareFinancial servicesGovernmentEducation
ResourcesDocsDetection catalogPosture check catalogChangelogBlog
CompanyTrustAboutCareersPressContact
Start your free Preview
Docs menu

At-Scale Reads: Cursor Pagination, NDJSON Streaming and Denormalized Edge Fields

How GraphnAI keeps large reads fast in big environments: denormalized edge fields that save lookups, cursor pagination, and reach data streamed level by level as NDJSON. For administrators and API clients.

Roles
Viewer (topology and reach views), Operator (inventory, detection alert and analysis views, and starting a full sync)
Updated
On this page

Overview

GraphnAI handles environments with hundreds of thousands of identities and tens of millions of edges. At that size, three kinds of read need care: a walk out from a high-degree node (an identity with hundreds of thousands of edges), a long list read page by page, and a large count or grouping. Three things keep them fast:

  • Denormalized edge fields. Each edge carries the type, criticality, tier, domain, account state and display name of both its ends. Counting and grouping read the edges alone, without fetching each identity.
  • Cursor pagination. Large lists page with signed cursors. A cursor seeks straight to the next page, so page 10,000 costs the same as page 1.
  • Streamed reach data. Reach graphs receive one JSON object per line as each level finishes. The first level arrives in seconds, and deeper levels follow.

The Identity Inventory and Detection Alerts pages stream their lists. Cursors serve API clients and the drill-down tables.

Prerequisites

  • Role: Viewer for the topology and reach views. Operator or Admin for the inventory, detection alert and analysis views. Operator to start a full sync.
  • Edge fields: A sync writes the edge fields. See Edge Fields and Upgrades.
  • No configuration: The cursor signing key comes from the install key that GraphnAI creates on its host at first start. See Cursor key changes.

Cursor Pagination

How it works

A paginated list returns an opaque string, the cursor, in nextCursor. Sending it back with the next request returns the next page. When nextCursor is null, there are no more rows.

A cursor is opaque. Its structure isn't documented and can change between releases. Don't parse it, build one, or keep it long term. Treat each cursor as a token for the next request.

Lists that page with cursors

List Sort
Identity inventory By risk (default), name, criticality, edge counts, permission count or membership counts, in either direction
Detection alerts Newest first
River drill-down tables Fixed
Edge tables behind a river line Heaviest first
Security Posture findings Fixed

Replay protection

The sort field and direction are signed into the cursor. A cursor from a list sorted by risk is refused when it's sent with a sort by name. A cursor from one list is refused on another. Filters such as type and domain aren't part of the cursor, so keep them the same from page to page.

Tampered cursor behavior

Cursors are signed with HMAC-SHA256. A cursor changed by even one character is refused with an "invalid cursor" error. The error doesn't say which part failed, so probing the check teaches nothing. When a client gets it, it starts again from the first page.

Cursor key changes

The cursor signing key comes from the install key, which GraphnAI creates on its host at first start and keeps in the server's data folder. You don't set it or rotate it. The key can be replaced, for example when you restore a different data volume. Every cursor signed with the old key then stops working, and every user is signed out. A client that was part way through a list gets an "invalid cursor" error on its next page, and starts again from the first page. There's no grace period.

A server restart with the same install key doesn't affect cursors.

NDJSON River Streaming

Overview

Reach data streams level by level as newline-delimited JSON (NDJSON). Each line of the response is one complete JSON object, called a frame. The connection stays open until every level is done or the client disconnects.

Each user can hold 4 reach streams open at once, and the server 32. Past that, the server refuses a new stream and asks the client to retry shortly.

Frame protocol

Every frame has a kind field. There are five kinds.

segment: One bucket per level

Sent once for each combination of level, source type, edge type and target type. It carries the data.

{
  "kind": "segment",
  "hop": 1,
  "fromType": "ServicePrincipal",
  "edgeType": "HasPermission",
  "toType": "Computer",
  "count": 245678,
  "criticalCount": 12,
  "highCount": 3245,
  "mediumCount": 142112,
  "tier0Count": 0
}

count is exact at every level.

hop-progress: A level still being counted

Sent while a level with a large number of identities is being counted. processed and frontierTotal say how far through the level's identities the count has got.

{"kind":"hop-progress","hop":2,"direction":"outbound","processed":250000,"frontierTotal":501459}

capped: A limit was reached, or the map is complete

Sent at most once per level and direction. reason says why:

reason Meaning
frontier-ceiling The level passed the size limit, 1,000,000 identities by default. Deeper levels still stream, built from the counted part.
depth-budget Mapping reached its time limit, 2 minutes by default. The levels already sent are right.
saturated The identity reaches at least 90% of the directory by default, so mapping stopped. This means the map is complete. 
{"kind":"capped","hop":2,"direction":"outbound","reason":"frontier-ceiling","limit":1000000,"totalTargets":1431207,"processed":1000000}

error: A level failed

Sent when a level's query fails. The connection then closes, and no data follows.

{
  "kind": "error",
  "hop": 3,
  "error": "hop processing failed"
}

complete: A clean end of stream

Sent as the last frame when every level finished without an error.

{
  "kind": "complete",
  "lastHop": 4
}

A stream that ends without a complete frame was cut short. The client disconnected, the server restarted, or the network dropped. The product shows "Loading stopped early, so the map is incomplete. Reload to try again." in that case.

Cancellation

When the client closes the connection, for example by leaving the page, the server stops the running query at its next batch and starts no new ones. An abandoned stream doesn't keep using the database.

Proxy and buffering considerations

Some reverse proxies hold the whole response before they pass it on, which defeats streaming. The server asks nginx not to buffer the stream. Configure any other proxy in front of GraphnAI to pass the stream through unbuffered. The product's reach graphs need the stream.

Server settings

An administrator of a very large environment can tune the stream through server settings: the size limit for one level (1,000,000 identities by default), the time limit for a whole stream (2 minutes by default), the share of the directory at which a reach counts as complete (90% by default), and how much work runs at once. They change server efficiency and when the partial notices appear.

Capped Response Envelope

Large lists add the same cap fields to every response, whether or not a cap was reached.

Envelope fields

Field Type Description
nextCursor string or null Cursor for the next page. null when there are no more pages.
capped boolean true when a server-side limit cut the result
capReason string or null Why the limit applied. null when capped is false.
limit integer The page size or limit that applied to this request

Cap reasons

Value Where Meaning What to do
max-rows Identity inventory and analysis lists The result passed the row limit. For the inventory, that's 100,000 rows. Narrow the filter by type, domain or preset
target-list-limit Access intelligence Each target list stops at the limit. The counts still cover every target. None needed
hop-limit Membership chain Each level of the chain stops at the limit, in a fixed order Investigate a large group directly
bfs-cap The older single-response reach data The deeper levels are a sample Use the streamed reach data for exact counts

Streamed reach data reports its limits in capped frames. See Frame protocol.

Understanding capped and nextCursor

The two fields are separate signals:

  • nextCursor set, capped false: more pages exist.
  • nextCursor null, capped false: there are no more rows.
  • capped true on the inventory: the result passed 100,000 rows. Every page says so, and pages stop at row 100,000. Narrow the filter to reach the rest.
  • capped true on an analysis list: more rows exist past this page.

Cap messages in the product

When a list or map reaches a cap, an amber line appears above it:

Reason Message
max-rows "Showing the first N. Narrow the filter to see the rest."
target-list-limit "Showing the first N targets in each list. The counts cover all of them."
frontier-ceiling "Counts cover N of M identities at this level. Deeper levels are partial."
depth-budget "Mapped N identities before the time limit. Deeper levels weren't expanded."
bfs-cap "Showing a partial map. Deeper levels are sampled."

Edge Fields and Upgrades

Denormalized edge fields that save lookups

Each edge carries twelve fields copied from its two ends when a sync writes it:

Field Type Description
fromType string Identity type of the source, such as User or ServicePrincipal
toType string Identity type of the target
fromCriticality integer Criticality of the source: 1 Low, 2 Medium, 3 High, 4 Critical. Absent when none.
toCriticality integer Criticality of the target
fromTier integer 0 when the source is Tier 0. Absent otherwise.
toTier integer 0 when the target is Tier 0. Absent otherwise.
fromDomain string Domain of the source
toDomain string Domain of the target
fromAccountEnabled boolean Whether the source account is enabled. Absent when the directory doesn't say, as for some Entra ID service principals.
toAccountEnabled boolean Same for the target
fromDisplayName string Display name of the source when the edge was written. The next full sync refreshes it.
toDisplayName string Display name of the target

With these fields, reach graphs, blast radius graphs and the Tier 0 credential exposure check read edges without fetching each identity. Edges from sessions and privileges carry them too.

The maxTargetCrit field

On inventory entries, maxTargetCrit is the highest criticality among the targets an identity reaches, as an integer from 0 to 4. It's 0 when none of its targets has a criticality. The Criticality Gap score in over-permission analytics counts those targets as 0 too.

After an upgrade

An upgrade converts the data GraphnAI already stores when the server starts. You don't reset the graph or run a command against the database.

When a release adds a value that only the directory can supply, the next scheduled sync collects it. The release notes say when that happens. To collect it sooner, run a full sync from Bridges.

On an environment of about 900,000 identities and 10 million edges, a full sync takes several hours. Wait for it to finish before you compare counts.

Troubleshooting

A list reports "invalid cursor"

  • Cause: The sort or direction changed since the cursor was issued, the cursor came from another list, or the install key was replaced. On a river drill-down table, a sync that changed edges also ends the cursor.
  • Fix: Start again from the first page. Don't save cursors to use later.

A reach graph shows nothing for a large identity

  • Cause: A river index is missing from the database, no sync has finished for the identity's domain yet, or the identity reaches nothing in the levels shown.
  • Fix: Restart the server, which creates missing indexes when it starts. If the graph is still empty, check the bridge's sync history for a finished sync of that domain. If there's none, run a full sync from Bridges.

The Criticality Gap is 0 for every identity

  • Cause: None of the targets has a criticality, so every target counts as 0 and every gap is 0.
  • Fix: Check the rules on Admin > Classification Rules, on the Criticality tab. After you change a rule, the gaps update by the end of the next full sync.

maxTargetCrit is 0

  • Cause: None of the identity's targets has a criticality.
  • Fix: None needed. If you meant every identity to get at least Low, check the rules on Admin > Classification Rules, on the Criticality tab.

A cap message appears on a list you expected to be complete

  • Cause: The request reached a server-side limit, for example a broad filter in an environment of 500,000 or more identities, or a reach from an identity with very many edges.
  • Fix:
    • For the inventory, narrow by type, domain or preset.
    • For a reach graph in Graph Explorer, pick one direction in the toolbar in place of both.
    • For detection alerts, filter by severity or date.

Reach graphs load all at once, or not at all, behind a proxy

  • Cause: A proxy between the browser and GraphnAI holds the stream until it ends.
  • Fix: Configure the proxy to pass the reach stream through unbuffered, for example by honoring the X-Accel-Buffering: no header the server sends.

    ↑↓ to move, Enter to open