Reduce size of Lighthouse payload

[rviscomi]

The latest lighthouse.2018_10_15 table is 237 GB. Querying all lighthouse tables currently costs 4.15 TB and runs in several minutes.

1. identify parts of the JSON payload that are unnecessary or unlikely to have analytical value and also significant contributors to the payload size
2. modify the Dataflow pipeline to omit these parts of the payload
3. profit

[connorjclark]

Can you point to where this trimming could be done?

[rviscomi]

Hey @connorjclark the get_lighthouse_reports function in the Dataflow pipeline would be the place where we can trim off excess response data

bigquery/dataflow/python/bigquery_import.py

Lines 188 to 222 in acef15a

	def get_lighthouse_reports(har):
	"""Parses Lighthouse results from a HAR object."""
	if not har:
	return
	report = har.get('_lighthouse')
	if not report:
	return
	page_url = get_page_url(har)
	if not page_url:
	logging.warning('Skipping lighthouse report: unable to get page URL (see preceding warning).')
	return
	# Omit large UGC.
	report.get('audits').get('screenshot-thumbnails', {}).get('details', {}).pop('items', None)
	try:
	report_json = to_json(report)
	except:
	logging.warning('Skipping Lighthouse report for "%s": unable to stringify as JSON.' % page_url)
	return
	report_size = len(report_json)
	if report_size > MAX_CONTENT_SIZE:
	logging.warning('Skipping Lighthouse report for "%s": Report size (%s) exceeded maximum content size of %s bytes.' % (page_url, report_size, MAX_CONTENT_SIZE))
	return
	return [{
	'url': page_url,
	'report': report_json
	}]