Skip to content

Postpaid Billing

Postpaid billing has two distinct execution modes — on-demand single-subscriber billing and full bill-cycle batch processing — sharing the same BillingHandler, the same activity queue, and the same database-side billing primitives. CDR ingestion (the upstream feed) runs as a separate role.

Activity Queue

Both billing modes are dispatched through the activities table. CRM writes activity rows; one or more billing processes drain them. The activity protobuf is:

message Activity {
    int32 id = 1;
    int32 type = 2;        // SINGLE_BILLING=1, FULL_BILLRUN=2, PREPAID_RATING=3
    string param = 3;       // MSISDN (single) or billcycle (full)
    string param2 = 4;      // month (full)
    string param3 = 5;      // year (full)
    string param4 = 6;      // reserved
    string param5 = 7;      // reserved
    string param6 = 8;      // reserved
    int32 status = 9;       // 0 pending, 99 processing, 1 completed, 2 error
    string error = 10;
}

BillingHandler::handle_timeout is invoked on each reactor tick (app.interval). It opens a DB_layer, calls get_new_activity() until the queue is empty, and dispatches each one onto the thread pool via processActivity(). Completed activities are closed back to the database with close_activity(); on success a Notification row is inserted so the operator sees the result on the CRM dashboard.

Per-Bill-Cycle Process Topology

The platform expects one server process per bill-cycle. The reference deployment ships four — billing1.toml … billing4.toml — each with a different billing.billcycle. Every billing process drains the same activity table, but processActivity rejects with ret = -2 when the activity's bill-cycle does not match the process configuration. This makes scaling linear: add a new billingN.toml per cycle, restart, and the workload partitions itself.

On-Demand Single-Subscriber Billing

Triggered by the CRM ExecuteBilling RPC, which writes an activity with type=SINGLE_BILLING and the MSISDN in param. The handler:

  1. Loads the subscriber by MSISDN. If subscriber.billcycle does not match the configured cycle for the running process, returns -2 (silently skipped — another process will pick it up).
  2. Generates a 32-character transaction ID via Utils::gen_transaction_id(). This becomes the bill's billing_id and is stamped on every CDR included in the bill.
  3. Calls BillingHandler::billOne(billing_id, subscriber_id, year=0, month=0, rater). Year and month default to zero, which signals all unbilled periods.
  4. billOne calls DB_layer::getUnbilledPeriods(sub) to discover every distinct (year, month) with unbilled CDRs, and walks each period producing one bill per period. When multiple periods are billed at once, the second and later periods carry a suffixed billing id (<base>_2, <base>_3, …) so the downstream PDF formatter can disambiguate them.
  5. After insert, every CDR with the matching billing_id is advanced to EVENT_STATUS_BILLED via updateEventStatusToBilled.
  6. A Notification is queued summarising success or failure for the operator UI.

If the caller passes an explicit year and month (rare — currently no path does), only that single period is billed.

Full Bill-Cycle Batch (Bill Run)

Triggered by ExecuteBillCycle, which writes an activity with type=FULL_BILLRUN, the bill-cycle number in param, the month in param2, and the year in param3. The handler validates the parameters before dispatching to the database:

  • param2 (month) must be present and in [1, 12].
  • February must not request day > 29.
  • param must equal the configured billing.billcycle for this process.

On any validation failure the activity is closed with status 2 and an error string; otherwise control passes to DB_layer::billCycle(activity, rater), which iterates every active postpaid subscriber on this bill-cycle, aggregates their CDRs for the requested month, computes per-service costs, and produces one billing::Bill row per subscriber. The bill row records the subscriber id, MSISDN, bill-cycle, month, year, billing id, four svcN_cost totals, any carried-over debt from prior unpaid bills, and the comment / URL fields used by the PDF layer.

Billing Handler Lifecycle

// main.cpp
BillingHandler billing_handler{};
if (UTILS::instance()->get_d("modules", "billing")) {
    billing_handler.setBillcycle(UTILS::instance()->get_d("billing", "billcycle"));
    ACE_Time_Value initial_delay{UTILS::instance()->get_d("app", "initial_delay")};
    ACE_Time_Value interval{UTILS::instance()->get_d("app", "interval")};
    ACE_Reactor::instance()->schedule_timer(&billing_handler, nullptr,
                                            initial_delay, interval);
}

In-flight workers are tracked through std::vector<std::future<void>>; on each tick, BillingHandler::handle_timeout drops any future whose state is ready and starts new ones. There is no explicit fan-out cap beyond the size of the thread pool, so back-pressure is implicit (the queue inside the pool grows if workers are slower than the activity rate).

Heartbeat and Watchdog

A long bill run can take minutes or even hours on modest hardware, but a crashed worker would otherwise leave its activity row stuck in PROCESSING forever. To distinguish "still working" from "stuck", every PROCESSING activity carries three timestamps:

  • started_at — UNIX seconds, stamped atomically with the PENDING → PROCESSING transition in DB_layer::get_new_activity.
  • heartbeat_at — refreshed inside the worker. For full bill runs, DB_layer::billCycle emits one heartbeat per app.activity_heartbeat_interval_subs subscribers (default 10). The cost of a heartbeat is one UPDATE-by-PK, so even on slow disks the overhead is bounded.
  • finished_at — stamped by close_activity alongside the terminal status, giving dashboards an authoritative wall-clock duration without scanning sys_update_date (which also fires on every heartbeat).

At the top of each tick, before the new-activity loop, BillingHandler::handle_timeout calls DB_layer::evictStaleActivities(stale_s). The bulk UPDATE transitions every PROCESSING row whose heartbeat_at is older than app.activity_heartbeat_stale_s seconds (default 600) to ACTIVITY_STATUS_ERROR with an explanatory error message stating the worker is presumed dead. Rows whose started_at is zero — for instance, hand-seeded test fixtures — are exempt from eviction by design. Setting activity_heartbeat_stale_s = 0 disables the watchdog entirely.

Mutual exclusion per bill-cycle

The activities table is the synchronisation point for bill-cycle work. A second FULL_BILLRUN request for a cycle that is already being processed must not raise a parallel worker — both runs would otherwise issue the same billing_id ranges and double-bill the affected subscribers. The guard lives inside DB_layer::get_new_activity itself: the SELECT FOR UPDATE clause is suffixed with a NOT EXISTS subquery that filters out any pending FULL_BILLRUN row whose param (the bill-cycle number) is also present in another row at status = PROCESSING. The gate is therefore atomic with the row claim — no race window between "check whether the cycle is busy" and "claim the row". The skipped pending row simply waits; the next tick re-evaluates it, and as soon as the predecessor closes (or the watchdog evicts it), the gate clears and the row is dispatched on the very next tick.

CDR Loader

The CDR loader (CDRLoader, in include/cdr_loader.hpp) is the upstream feed for postpaid billing. It is run as its own role (cdr_loader.toml) so that file I/O and rating cannot starve the billing or DIAMETER paths.

Directory layout

TOML key Default Purpose
cdr.new_cdr_path cdr/new Polled directory; arriving files are picked up here.
cdr.processed_cdr_path cdr/processed Successful files are moved here with the .done suffix appended.
cdr.error_cdr_path cdr/error Failed files (deserialise / validate / DB insert) are moved here with the .error suffix.
cdr.duplicate_cdr_path cdr/duplicate Files whose CDR id is already present in offline are moved here with the .duplicate suffix.
cdr.new_cdr_extension .cdr File extension required for inclusion in the scan.

Processing

handle_timeout enqueues a single scan task on the thread pool, guarded by std::atomic<bool> is_processing. Each file is loaded into a CDR wrapper, deserialised via Protobuf's ParseFromIstream, validated (A-number resolves to a known subscriber, service id is in range), rated via Rater::pre_rate → rate → post_rate, and inserted via DB_layer::insert(billing::CDR). Errors at any step move the file to error/ so the source remains traceable for offline diagnosis. A successful file is renamed to <original>.done in processed/.

Idempotent ingest

The protobuf CDR.id is the upstream mediation id and is persisted on the offline row as the mediation_id column, which carries a UNIQUE constraint. Re-presenting a CDR is therefore not an error: after validate(), the loader calls DB_layer::cdrExists(mediation_id) and skips the rate-and-insert pipeline when the id is already present, moving the source file to cdr/duplicate/<name>.duplicate instead of cdr/error/. As a safety net, the INSERT itself catches the duplicate-key error from the X DevAPI driver and routes through the same path, closing the race window when two threads see the same file at the same time. CDRs whose id is 0 carry no upstream identifier and are not deduplicated — the mediation_id column is NULL for those rows, and MySQL's UNIQUE semantics allow that to repeat. The synthetic CDR generator stamps a monotonically-increasing id derived from the current epoch so dev/test traffic exercises the dedup path naturally.

Every duplicate rejection writes a notifications row tagged source='cdr_loader' so the operator dashboard surfaces the event with a deep-link to the usage page.

CDR generator

bin/cdr_generator -i <toml> -n <count> -o <output_dir> writes synthetic CDR files for development and load testing. It draws from active postpaid subscribers in the database, randomises service id, units (within gen_voice_min/max and gen_data_min/max), and event timestamp (within gen_date_start … gen_date_end), and serialises one protobuf per file using the same schema the loader consumes. See Configuration for the generator-specific TOML keys.

Bill-Cycle Reference Data

REF_Billcycle carries one row per cycle and a day column for the cut-off day of the month. BillingHandler::billCycle consults REFDATA::instance()->getBC() for sanity checks (e.g. February day > 29 detection), and DB_layer::billCycle uses the day field to compute the closing window for which CDRs are eligible for the cycle.

Status Transitions

A CDR moves through three statuses on the postpaid path:

  • EVENT_STATUS_PENDING (1) — newly inserted, not yet rated. Used in some test paths only; the loader rates inline.
  • EVENT_STATUS_RATED (2) — rated, ready to be picked up by the next bill-cycle run.
  • EVENT_STATUS_BILLED (4) — included in a bill. Stamped by updateEventStatusToBilled(billing_id).

EVENT_STATUS_ERROR (3) is reserved for downstream rating failures observed during mass rating; the file-based loader does not use it (failed files are moved to error/, not inserted).

Bill Statuses

A billing::Bill carries a status_id that walks through:

  • BILL_STATUS_NEW (1) — fresh, awaiting PDF rendering.
  • BILL_STATUS_PROCESSING (2) — locked by a bill-formatter worker.
  • BILL_STATUS_PAID (5) — set after PDF render for prepaid bills (which are settled at the moment of issue).
  • BILL_STATUS_UNPAID (3) — set after PDF render for postpaid bills (awaiting payment).
  • BILL_STATUS_ERROR (4) — formatter raised an error.

The transition from billing to invoicing is therefore new → processing → paid|unpaid|error, owned end-to-end by the bill formatter. See Invoicing.