Roadmap 0.10, Performance roadmap — wave 3

PERF.44 Website: session_activity and analytics_visitors hold 1.6 M rows and 1,015 MB (83 % of the database), 42 % bots, no retention, one table never read

  • Platform
  • Planned

Measured: session_activity 1.02 M rows / 646 MB growing 51,086 rows a day (21,675 of them is_bot = 1), analytics_visitors 597k rows / 369 MB growing 38,644 a day because a cookie-less client gets a fresh visitor id on every request; the table is written on every page view (Analytics.php:75) and read nowhere. Nothing prunes either (no DELETE in bin/ or the classes except the smoke test's own rows). The report queries full-scan the table (EXPLAIN ANALYZE: summary 1.8 s, topPages 2.4 s, each audience group 1.7 s, tryOpens 1.3 s), so the nightly warm job is roughly 60 to 80 s of scans that push the whole table through a 1 GB server-wide buffer pool these two tables alone fill. CadCraft analytics is about 1.1 inserts a second, over half of all INSERTs on this MySQL, each an fsync (two autocommits per page view). At 10x the warm job takes ten minutes and evicts every other database on the server. Fix: (1) do not store bot page views as rows (count per day per bot name in a small table) and give a visitor with no cc_vid cookie no analytics_visitors row until the second request; (2) retention for humans (owner's number; 30 days for any bot rows kept) as monthly partitions by TO_DAYS(created_at) dropped with ALTER … DROP PARTITION, or a nightly batched DELETE on idx_created; (3) a daily rollup table (day, url, country, region, city, device, os, browser, referrer host, campaign, lang, is_bot, views, people) filled for yesterday only, so the dashboard and reports read at most 365 rows per group; (4) stop writing analytics_visitors or read it somewhere; drop the unused idx_session. Keep the privacy page true. Combines the SQL and server tracks (sql-audit.md 4, server-api.md 9).

Waits on: Owner: how long to keep page-view rows (400 days keeps the year-over-year comparison the analytics page offers).

Add a comment

Community comments

No comments yet

Add detail as we go: how you would use it, files or steps that show the problem, what other programs do. The team reads every comment.

Sign in or create a free account to add a comment.