Our fork of Guthix2827/osrs-market - a C++ backend and React frontend for collecting, storing and visualizing OSRS market data
  • TypeScript 46.2%
  • C++ 33.3%
  • CSS 14.6%
  • Shell 4%
  • Python 1%
  • Other 0.8%
Find a file
Repository files (latest commit first)
Filename Latest commit message Latest commit date
2026-10-22 04:15:51 +03:00
.github Create FUNDING.yml 2026-08-21 19:23:08 +03:00
auth fix: auth creation admin on kickstar 2026-08-24 10:45:37 +03:00
auth-service Improve CORS security, database transaction safety, and item history loading 2026-08-27 10:04:11 +03:00
backend perf(db): optimize latest item price lookup using LATERAL joins 2026-10-22 02:54:40 +03:00
docs readme update & screenshot 2026-08-23 22:02:45 +03:00
frontend grid style fix due to removed dailyVolume 2026-10-22 04:15:51 +03:00
scripts feat(auth): add secure production FusionAuth bootstrap 2026-08-24 09:28:00 +03:00
.dockerignore feat(auth): integrate FusionAuth with secure Node.js sessions 2026-08-24 08:07:33 +03:00
.env.example Improve CORS security, database transaction safety, and item history loading 2026-08-27 10:04:11 +03:00
.gitignore feat(auth): add secure production FusionAuth bootstrap 2026-08-24 09:28:00 +03:00
docker-compose.dev.yml Improve CORS security, database transaction safety, and item history loading 2026-08-27 10:04:11 +03:00
docker-compose.prod.yml Fix production auth secrets and Market Movers proxy routing 2026-08-27 10:33:54 +03:00
LICENSE update README, screenshots, and licensing 2026-08-18 00:24:35 +03:00
README.md docs: add production operations and troubleshooting commands 2026-10-22 02:30:15 +03:00

OSRS Market

OSRS Market is a full-stack Old School RuneScape market data project built as a practical exercise in modern C++ backend development.

The application collects item metadata and real-time Grand Exchange price data from the RuneScape Wiki APIs, persists historical market information, and exposes the data through its own API for a React frontend.

The goal is to build a fast, self-contained market data service that does not require the frontend to depend directly on the upstream API.

Screenshots

▶ Watch the 15-second item page demo OSRS Market item page

Project Status

OSRS Market is under active development. The core market-data pipeline, historical price system, authentication layer, market discovery features, and responsive frontend are working end to end.

Implemented

Backend & Data Pipeline

  • C++20 backend built with Crow
  • PostgreSQL persistence with incremental SQL migrations
  • RuneScape Wiki item mapping synchronization
  • Local item icon caching
  • Item metadata revision history
  • In-memory metadata and latest-price stores
  • Bulk five-minute market price collection
  • Change-based price event persistence
  • Background price collection and mapping refresh workers
  • Historical timeseries API
  • Coverage-aware history caching that avoids caching incomplete ranges
  • 24-hour, 7-day, 30-day, and 1-year history ranges
  • Background historical backfill for missing ranges
  • Historical backfill state tracked independently from actual trade coverage
  • Wiki-provided timestamps preserved during historical backfill
  • Support for sparse, new, and rarely traded items without repeatedly downloading completed historical ranges
  • Watch-summary API backed by the shared PriceHistoryCache
  • Serialized database access to prevent overlapping transactions on shared libpqxx connections

Frontend

  • React + TypeScript frontend integrated with the C++ API
  • Item search by name or ID
  • Interactive price and volume charts
  • Synchronized chart zoom across price, volume, and trade-distribution views
  • Bought vs sold volume visualization
  • Market statistics including margin, ROI, limit profit, price change, spread, volume, and liquidity
  • Responsive desktop, tablet, and mobile layouts
  • Sticky responsive navigation and search
  • About page and site footer
  • Latest instant buy/sell prices refreshed independently from historical chart data

History Loading & Live Updates

  • Per-range client-side history state for fast chart switching
  • Previously loaded ranges are reused while their data is still fresh
  • 24-hour history freshness follows five-minute market update boundaries
  • 7-day history uses hourly freshness boundaries
  • 30-day and 1-year history use daily freshness boundaries
  • 24-hour charts automatically refresh shortly after each five-minute market interval
  • Latest instant buy/sell prices refresh every minute
  • Changing historical ranges does not unnecessarily reload latest-price data
  • Navigating to another item resets the chart to 24H and loads fresh history for that item

Watchlist

  • Persistent client-side watchlist
  • Slide-out watchlist drawer
  • Drag-and-drop item ordering
  • Per-item comparison range selection
  • Midpoint price tracking with 30m, 1h, 6h, 12h, and 24h comparisons
  • Per-item price tracking from the moment an item is added
  • Watched-since marker on the price history chart
  • Per-item watch-summary refresh scheduling
  • localStorage persistence

Market Discovery

  • Market Movers page
  • Top gainers and losers
  • Highest-volume and highest-value rankings
  • Daily market snapshot cards
  • Responsive mobile market rows
  • Daily cached Market Movers backend responses

Authentication

  • FusionAuth-based user authentication
  • Dedicated authentication service
  • Session-based frontend authentication flow
  • Login, callback, session, and logout endpoints
  • Development and production authentication configuration
  • Local secret management without committing credentials to the repository
  • C++ backend remains independent from authentication UI concerns for public market endpoints

Security & Infrastructure

  • Explicit CORS origin configuration through FRONTEND_ORIGIN
  • Restricted production frontend origins instead of wildcard CORS
  • Docker development environment
  • Production Docker Compose stack
  • Nginx frontend and API reverse proxy
  • Service health checks
  • PostgreSQL first-run schema initialization from backend/sql
  • Backend SQL migration runner for incremental schema updates
  • Non-blocking startup workers for mapping refresh, icons, price collection, gap recovery, and historical backfill
  • Standalone backfill-all mode for repairing historical market data

Known Limitations / In Progress

  • Startup gap detection currently uses the latest global market timestamp and can miss per-item gaps when newer records already exist. Full historical backfill can be used to repair these gaps while per-item recovery is improved.
  • Authentication is implemented, but user-owned features such as server-side watchlist persistence and premium account functionality are still being integrated.
  • Historical item lifecycle boundaries are not yet enforced, so additional validation is planned for new, removed, temporary, and reused item IDs.

Planned / Next Steps

  • Persist watchlists to authenticated user accounts
  • Admin functionality protected by authentication
  • Social sign-in providers such as Google and Apple
  • RuneLite integration for opt-in synchronization of completed Grand Exchange trades
  • Verified merchant portfolios with profit, ROI, risk, and performance tracking
  • Merchant hiscores ranked by verified flipping and merchandising performance
  • Virtual GP investments in public merchant portfolios, with no connection to real GP or real money
  • Share functionality
  • Additional market statistics, aggregation, and market-trend analysis
  • Price-analysis and experimental prediction features
  • WebSocket live price updates for actively viewed items
  • Further frontend and backend performance hardening
  • Historical price retention and rollups:
    • retain high-resolution five-minute data for the recent 24-hour window
    • aggregate older data into hourly history
    • archive raw events for future analytics
    • remove expired high-resolution rows from the live database
  • Item lifecycle validation:
    • detect and store an item's official release date or earliest valid market date
    • reject historical records from before an item's valid market lifetime
    • protect against reused item IDs previously associated with League, Deadman Mode, beta, temporary, or removed content

Stack

Backend

  • C++20
  • Crow
  • CPR / libcurl
  • nlohmann/json
  • libpqxx
  • PostgreSQL
  • CMake
  • Ninja

Frontend

  • React
  • TypeScript
  • Vite
  • Recharts

Infrastructure

  • Docker
  • Docker Compose
  • PostgreSQL
  • Git / GitHub

Architecture

The backend acts as an intermediary between the RuneScape Wiki price APIs and the frontend.

                   RuneScape Wiki APIs
                          |
             +------------+------------+
             |                         |
         Item Mapping                5m Prices
             |                         |
             v                         v
     MappingRefreshJob       FiveMinutePriceRefreshJob
             |                         |
      +------+------+            +-----+------+
      |             |            |            |
      v             v            v            v
 PostgreSQL    MappingStore  LatestPriceStore PostgreSQL
                  (RAM)          (RAM)       price_events
      |
      v
 Item revisions

                  Backend API
                       |
                       v
                 React Frontend

The frontend does not need to communicate directly with the RuneScape Wiki APIs.

This allows the backend to continue serving previously collected data if the upstream API is temporarily unavailable.


Item Metadata

Item metadata is obtained from the RuneScape Wiki mapping API.

Examples of metadata include:

  • Item ID
  • Name
  • Examine text
  • Members status
  • Store value
  • High alchemy value
  • Low alchemy value
  • Grand Exchange buy limit
  • Icon filename

Metadata is persisted in PostgreSQL.

Item Revisions

Item metadata can change over time.

Instead of overwriting the existing item information, the backend stores revisions.

items
  |
  +-- current_revision_id
          |
          v
    item_revisions

For example:

Item 6739
Dragon axe

Revision 1
    name: Dragon axe
    examine: Old description

Revision 2
    name: Dragon axe
    examine: New description

Only a changed item creates a new revision.

This preserves historical metadata while allowing the application to efficiently reference the current revision.


Item Icons

Item icons are downloaded from the Old School RuneScape Wiki and stored locally.

They are exposed by the backend using URLs such as:

/icons/6739.png

The item API therefore returns:

{
    "id": 6739,
    "name": "Dragon axe",
    "icon": "/icons/6739.png"
}

rather than requiring the frontend to know the upstream Wiki image URL.

Downloaded icons are stored in a Docker volume and survive container rebuilds.

Known missing icons are temporarily remembered to prevent repeatedly requesting the same unavailable resource.


Price Collection

The backend uses the RuneScape Wiki bulk 5-minute price endpoint.

/api/v2/osrs/5m

One request provides market information for many items.

This is significantly more efficient than requesting every item individually.

A normalized price point contains:

itemId
timestamp
avgHighPrice
avgLowPrice
highPriceVolume
lowPriceVolume

Prices can be null when no corresponding trade occurred during the interval.


Change-Based Price History

The backend does not blindly store every observation.

Instead, the latest known market state for each item is held in memory.

Wiki /5m
    |
    v
LatestPriceStore
    |
    +-- unchanged --> ignore
    |
    +-- changed ----> price_events

A price event is considered changed when one or more of these values changes:

avgHighPrice
avgLowPrice
highPriceVolume
lowPriceVolume

This avoids storing unnecessary duplicate observations for inactive items.

PostgreSQL additionally enforces uniqueness on:

(item_id, timestamp)

so the same 5-minute event cannot accidentally be inserted twice.


Database

The application currently uses PostgreSQL for persistent storage.

Main tables:

items
item_revisions
price_events

items

Stores stable item identities and points to the current metadata revision.

item_revisions

Stores historical versions of item metadata.

price_events

Stores historical market changes.

The primary history access pattern is:

WHERE item_id = ?
AND timestamp >= ?
ORDER BY timestamp

The table therefore has an index based on:

(item_id, timestamp)

Historical storage may later be partitioned or aggregated as the dataset grows.


Market Movers

Added a new Market Movers page for discovering high-activity and high-change Grand Exchange items.

Current features:

  • Top 30 market movers table
  • Range filters: 24H, 7D, 30D, 90D
  • Categories:
    • Top Gainers
    • Top Losers
    • Most Traded
    • Highest Value
  • Daily snapshot cards:
    • Biggest rise
    • Biggest fall
    • Highest volume
  • Responsive mobile layout with expandable item rows
  • Browser-side daily cache to reduce repeat API calls

Backend support:

  • GET /api/market/movers?range=24h&category=gainers
  • GET /api/market/movers/snapshot
  • Market mover results are precomputed and cached in memory
  • Cache refresh runs on backend startup and once daily after UTC midnight
  • Responses include daily Cache-Control headers for proxy/browser caching

Database Migrations

Fresh PostgreSQL volumes are initialized from:

backend/sql

Incremental schema changes after the initial database setup belong in:

backend/migrations

The backend runs pending migration files on startup before repositories, caches, workers, or Crow routes are initialized.

Applied migrations are recorded in schema_migrations, so each file runs only once.

The schema_migrations table is part of the base SQL initialization for new databases.


In-Memory Stores

PostgreSQL is the durable source of data, but frequently accessed current state is also kept in memory.

There are currently two major in-memory stores.

MappingStore

Contains the current item metadata.

Conceptually:

std::unordered_map<ItemId, ItemMapping>

This allows item API requests to avoid querying PostgreSQL for every request.

LatestPriceStore

Contains the newest known price state for each item.

Conceptually:

std::unordered_map<ItemId, PricePoint>

It is used to detect market changes before writing historical events.

This store will also eventually provide the current state for WebSocket updates.


Backend Startup Flow

The backend intentionally restores persistent state before starting its background collectors.

The startup process is approximately:

main()
 |
 +--> Read DATABASE_URL
 |
 +--> Run pending database migrations
 |
 +--> Connect to PostgreSQL
 |      |
 |      +--> item database connection
 |      |
 |      +--> price database connection
 |
 +--> Create repositories
 |      |
 |      +--> ItemRepository
 |      |
 |      +--> PriceRepository
 |
 +--> Hydrate LatestPriceStore
 |      |
 |      +--> Load latest price event for each item
 |      |
 |      +--> Store latest states in RAM
 |
 +--> Hydrate MappingStore
 |      |
 |      +--> Load current item revisions
 |      |
 |      +--> Store current item metadata in RAM
 |
 +--> Launch background workers
        |
        +--> MappingRefreshJob
        |
        +--> GapRecoveryJob
        |
        +--> IconDownloadWorker
        |
        +--> FiveMinutePriceRefreshJob
 |
 +--> Start Crow HTTP server

The HTTP server serves from the cached PostgreSQL state while the fresh item mapping sync runs in the background. When the sync completes, it replaces MappingStore.

This keeps Crow responsive during local development and production restarts even when item synchronization takes time.

The backfill-all command is different: it runs the mapping refresh synchronously before full history backfill because the backfill job needs a populated MappingStore.

backfill-all
 |
 +--> Refresh item mapping
 |
 +--> Run FullHistoryBackfillJob
 |
 +--> Exit

Why Price State Is Hydrated First

LatestPriceStore is an in-memory cache.

Without restoring it from PostgreSQL, every backend restart would make the first /5m response appear to contain completely new data.

For example:

Backend stops

RAM:
    empty

Backend starts

GET /5m
    |
    v
1882 items appear "new"

The database uniqueness constraint would prevent duplicate rows, but this would still cause thousands of unnecessary insert attempts.

Instead:

Backend starts
    |
    v
PostgreSQL
    |
    v
load latest event per item
    |
    v
LatestPriceStore
    |
    v
GET /5m
    |
    v
compare against known state

Only genuinely changed market states are then written.


Runtime Price Flow

Once startup is complete, price collection runs independently in the background.

FiveMinutePriceRefreshJob
          |
          v
 GET RuneScape Wiki /5m
          |
          v
      parse JSON
          |
          v
     ItemFilter
          |
          v
  LatestPriceStore
          |
     +----+----+
     |         |
 unchanged   changed
     |         |
   ignore      v
          PriceRepository
                |
                v
           PostgreSQL
           price_events

The refresh job currently runs periodically in its own thread.

The HTTP server therefore does not need to wait for upstream RuneScape Wiki requests when serving normal requests.


Runtime Mapping Flow

Item metadata synchronization is handled separately.

MappingRefreshJob
       |
       v
RuneScape Wiki /mapping
       |
       v
    ItemFilter
       |
       +--------------------+
       |                    |
       v                    v
ItemRepository         Icon queue
       |                    |
       v                    v
 PostgreSQL       IconDownloadWorker
       |
       v
revision detection
       |
       v
 MappingStore
     (RAM)

If metadata has not changed, no new revision is created.

If metadata changes, a new revision is inserted and the item's current_revision_id is updated.


Database Connections and Concurrency

Background jobs execute concurrently.

A PostgreSQL connection cannot have multiple active transactions simultaneously, so independent workers do not share the same libpqxx connection.

Conceptually:

Mapping worker
      |
      v
ItemRepository
      |
      v
PostgreSQL connection #1


Price worker
      |
      v
PriceRepository
      |
      v
PostgreSQL connection #2

A connection pool may replace this arrangement later as the number of concurrent database workloads grows.


Historical Data Flow

The bulk /5m endpoint is used for ongoing market collection.

Historical timeseries endpoints are used for initial history population and recovery after collector downtime.

On startup, the backend checks the newest timestamp stored in price_events.

Backend starts
      |
      v
Check MAX(price_events.timestamp)
      |
      +--> <= 10 minutes --> no recovery
      |
      +--> <= 24 hours --> recover using 24h history
      |
      +--> <= 7 days --> recover using 7d history
      |
      +--> > 7 days --> recover using 1y history

When a gap is detected, GapRecoveryJob runs in the background so recovery does not block the HTTP server.

Normal history requests are served from local PostgreSQL data and do not trigger per-item backfills.

A separate backfill-all command is available for initial database population or a complete historical synchronization.

After history has been populated, the continuous /5m collector extends it over time.


Frontend History Refresh

The frontend loads historical data from the backend and keeps previously visited ranges in a client-side cache. Each range owns its own dataset, avoiding unnecessary chart reprocessing when switching between 24H, 7D, 30D, and 1Y views.

The currently selected range is refreshed periodically. Stale requests can be cancelled when navigation changes, while previously loaded ranges remain immediately available in memory.

Real-time WebSocket delivery is not currently required; the existing periodic refresh is intentionally kept simple unless future product requirements justify push-based updates.



Watchlist

The frontend includes a persistent watchlist drawer for quick access to frequently viewed items.

Each watched item stores lightweight client-side state, including the selected comparison range and the most recent watch-summary response. The list is persisted in localStorage, so item order, selected ranges, and cached summaries survive page refreshes.

Supported comparison ranges:

30m
1h
6h
12h
24h

The watchlist displays the current market midpoint, calculated from the instant buy and instant sell prices.

Two price-change measurements are shown for each watched item:

  • Selected range — compares the current midpoint with the historical midpoint at the start of the selected 30m, 1h, 6h, 12h, or 24h range.
  • Since added — compares the current midpoint with the midpoint captured when the item was added to the watchlist.

The initial midpoint and watch timestamp are stored with the watched item, allowing the frontend to preserve the original tracking point while current watch-summary data continues to refresh.

When the watch timestamp falls within the visible price-history range, the price chart displays a vertical marker showing when tracking began.

These percentages represent market price movement and do not include Grand Exchange tax.

Watch-summary data is served by the backend using the existing 24-hour PriceHistoryCache. On a cache hit, no PostgreSQL history query is required. On a cache miss, the backend loads the 24-hour history from PostgreSQL, stores it in the shared cache when coverage is complete, and derives the closest available reference prices for the supported ranges.

Conceptually:

Watch item
   |
   v
GET /api/items/<id>/watch-summary
   |
   v
PriceHistoryCache (24h)
   |
   +-- hit  --> derive reference prices
   |
   +-- miss --> PostgreSQL --> cache --> derive reference prices
   |
   v
30m / 1h / 6h / 12h / 24h references
   |
   v
React watchlist drawer

The client refresh scheduler tracks freshness per watched item. Rather than polling the entire list continuously, it schedules a single timeout for the closest next refresh time and only updates items whose summaries are stale. Returning to a previously backgrounded browser tab also triggers a stale-item check.


Development

Start the development environment:

docker compose up

Build the C++ backend inside the development container:

docker compose exec backend cmake --build build

Restart the backend:

docker compose restart backend

Follow backend logs:

docker compose logs -f backend

Recent logs only:

docker compose logs --since=30s backend

Production

The production stack runs as:

Browser
  |
  | HTTPS (public domain)
  v
Reverse Proxy (optional)
  |  TLS termination / domain routing
  |
  | HTTP -> frontend exposed port
  v
Frontend Container
  |  Nginx :80
  |  - serves React static files
  |  - proxies /api/* and /icons/* to backend
  |
  | Docker internal network
  v
Backend Container
  |  Crow :8080
  |
  | Docker internal network
  v
PostgreSQL Container
     PostgreSQL :5432

The reverse proxy is optional. Without one, expose the frontend service directly through a host port in docker-compose.prod.yml.

The backend and PostgreSQL services do not need to be publicly exposed; communication between the frontend, backend, and database occurs over the internal Docker Compose network.

Docker health checks are configured for PostgreSQL, the backend API, and the frontend. The frontend waits for the backend to become healthy, while the backend waits for PostgreSQL.

On a fresh PostgreSQL volume, SQL files from backend/sql are mounted into /docker-entrypoint-initdb.d and executed automatically during first-time database initialization. Existing initialized volumes are left unchanged by PostgreSQL init scripts.

Incremental schema changes after first initialization are handled by the backend migration runner using SQL files from backend/migrations.

Development

Build

docker compose -f docker-compose.dev.yml build

Start existing containers

docker compose -f docker-compose.dev.yml start

Stop containers

docker compose -f docker-compose.dev.yml stop

Create and start

docker compose -f docker-compose.dev.yml up -d

Build and start

docker compose -f docker-compose.dev.yml up -d --build

Stop and remove containers

docker compose -f docker-compose.dev.yml down

Status

docker compose -f docker-compose.dev.yml ps

Backend logs

docker compose -f docker-compose.dev.yml logs -f backend

Frontend logs

docker compose -f docker-compose.dev.yml logs -f frontend

PostgreSQL logs

docker compose -f docker-compose.dev.yml logs -f postgres

Rebuild C++ backend inside running container

docker compose -f docker-compose.dev.yml exec backend cmake --build build

Restart backend

docker compose -f docker-compose.dev.yml restart backend

Production

Build

docker compose \
  --env-file .env.production \
  -f docker-compose.prod.yml \
  build

Start

docker compose \
  --env-file .env.production \
  -f docker-compose.prod.yml \
  up -d

Stop

docker compose \
  --env-file .env.production \
  -f docker-compose.prod.yml \
  down

Build and start

docker compose \
  --env-file .env.production \
  -f docker-compose.prod.yml \
  up -d --build

Status

docker compose \
  --env-file .env.production \
  -f docker-compose.prod.yml \
  ps

Backend logs

docker compose \
  --env-file .env.production \
  -f docker-compose.prod.yml \
  logs -f backend

API

Current example item endpoint:

GET /api/items/6739

Example response:

{
    "buyLimit": 40,
    "examine": "A very powerful axe.",
    "highAlch": 33000,
    "icon": "/icons/6739.png",
    "id": 6739,
    "lowAlch": 22000,
    "members": true,
    "name": "Dragon axe",
    "value": 55000
}

Item icons:

GET /icons/6739.png

Historical price data:

GET /api/items/6739/history?range=24h
GET /api/items/6739/history?range=7d
GET /api/items/6739/history?range=30d
GET /api/items/6739/history?range=1y

Item search:

GET /api/items/search?q=dragon

Watchlist price-change summary:

GET /api/items/6739/watch-summary

Example response:

{
  "itemId": 6739,
  "generatedAt": 1787262696,
  "currentMidPrice": 6912450,
  "references": {
    "30m": 6852761,
    "1h": 6951140,
    "6h": 6970347,
    "12h": 6875915,
    "24h": 6767910
  }
}

The reference values are historical midpoint prices derived from the closest available points in the cached 24-hour history. currentMidPrice represents the current market midpoint used by the frontend for range and watchlist price-change calculations.

Historical requests are served entirely from local PostgreSQL data and do not trigger any backfill or background work. Historical recovery is handled independently by GapRecoveryJob, which checks for gaps when the backend starts.


Development & Maintenance Commands

Build and start the backend

Build the backend image:

docker compose build backend

Start the backend service:

docker compose up -d backend

Rebuild during development

Recompile the C++ backend inside the running development container and restart the service:

docker compose exec backend cmake --build build && \
docker compose restart backend

Check item backfill status

View the completed historical backfills for a specific item:

docker compose exec postgres \
  psql -U osrs_market -d osrs_market \
  -c "
    SELECT
        item_id,
        lookback,
        last_completed_at
    FROM price_backfill_state
    WHERE item_id = 6739
    ORDER BY lookback;
  "

Replace 6739 with the desired item ID.

Full history synchronization

Run the one-time full historical backfill for all tracked items:

production

docker compose \
  --env-file .env.production \
  -f docker-compose.prod.yml \
  run -d --rm \
  --name osrs-backfill \
  backend \
  ./build/osrs-market-backend backfill-all

dev

docker compose \
  -f docker-compose.dev.yml \
  run -d --rm \
  --name osrs-backfill \
  backend \
  ./build/osrs-market-backend backfill-all

The temporary backend container is removed after the command completes. PostgreSQL data is persisted separately and is not removed. Normal startup gap recovery does not require this command.

Monitor full backfill progress

Check how many items have completed each historical lookback:

docker compose exec postgres \
  psql -U osrs_market -d osrs_market \
  -c "
    SELECT
        lookback,
        COUNT(*) AS completed,
        ROUND(COUNT(*) * 100.0 / 4648, 2) AS percent
    FROM price_backfill_state
    GROUP BY lookback
    ORDER BY lookback;
  "

4648 is the current number of tracked items. Update this value if the mapping size changes.


Data Source

Market and item data are sourced from the RuneScape Wiki / RuneLite real-time price APIs.

This project is not affiliated with Jagex, RuneScape, RuneLite, or the RuneScape Wiki.

Old School RuneScape and RuneScape are trademarks of Jagex Ltd.


Production Service Management

Production Docker Compose commands must load both environment files:

  • .env.production — production configuration
  • .secrets/production.env — sensitive credentials and authentication configuration

Run commands from the repository root.

Recreate the backend

Useful when the backend needs to reload its in-memory item mapping, recover from an unhealthy state, or restart its background workers.

docker compose \
  --env-file .env.production \
  --env-file .secrets/production.env \
  -f docker-compose.prod.yml \
  up -d --force-recreate backend

Recreate the authentication service

Useful when authentication configuration has changed or the authentication container is unhealthy.

docker compose \
  --env-file .env.production \
  --env-file .secrets/production.env \
  -f docker-compose.prod.yml \
  up -d --force-recreate auth

Rebuild and deploy the backend

Required when deploying changes to the C++ backend code or Docker image.

docker compose \
  --env-file .env.production \
  --env-file .secrets/production.env \
  -f docker-compose.prod.yml \
  up -d --build backend

Check service status

docker compose \
  --env-file .env.production \
  --env-file .secrets/production.env \
  -f docker-compose.prod.yml \
  ps

Follow backend logs

docker compose \
  --env-file .env.production \
  -f docker-compose.prod.yml \
  logs -f --tail=100 backend

Troubleshooting newly added items

When newly added RuneScape items are missing or display incomplete information:

  1. Inspect backend logs for mapping refresh failures or database foreign-key errors.
  2. Confirm the item exists in the items table and has a current entry in item_revisions.
  3. Check whether the backend mapping refresh has completed.
  4. Recreate the backend if its in-memory state appears stale.
  5. Verify the item's price history against the upstream RuneScape Wiki API before modifying historical records.

Important: Recreating a service does not rebuild its Docker image. Use --build when deploying code changes. Do not run backfill-all as a general-purpose restart or assume it repairs invalid upstream historical data.


Project Goals

This project is primarily being developed to explore and practice:

  • Modern C++ application architecture
  • HTTP services in C++
  • Concurrent background workers
  • Background recovery jobs
  • PostgreSQL from C++
  • Time-series market data
  • In-memory caching
  • REST API design
  • Dockerized C++ development
  • Production deployment
  • React/C++ integration
  • Responsive frontend architecture
  • Client-side caching and chart performance

License

This project is source-available for educational, portfolio, and security-review purposes.

Copyright © 2026. All rights reserved.

See LICENSE for details.