Description of Methodology (DOM)
© 2018 Hulu, LLC | all rights reserved
Hulu, LLC | 2500 Broadway, 2nd Floor | Santa Monica, CA 90404 | (310) 571-4700 | hulu.com
1.4 | last updated September 6, 2018
Frequency of Updates
Owner of Update Process
No part of this publication may be reproduced, transmitted, transcribed, stored in a retrieval system, translated into any other language in any form or by any means, electronic or mechanical, including photocopying or recording, for any purpose, without the express permission of Hulu, LLC.
This document is provided for informational purposes. The information is provided “as is” and without warranties of any kind. Due to continuing product development, the information is subject to change without notice. Hulu, LLC shall not be responsible or liable for any inaccuracies, errors, or outdated information.
This document is intended for advertisers. It provides a summary of the video-streaming measurement processes employed by Hulu, LLC (Hulu). It includes a general description of Hulu’s measurement methodology, filtration processes and reporting procedures. Issuance of this document is in accordance with Hulu’s accreditation conferred by the Media Rating Council (MRC).
Hulu provides an Ad-supported streaming digital video service that offers a selection of TV shows, clips and movies via its video players at hulu.com, embedded Hulu video players hosted at other internet sites, mobile devices via the Hulu mobile application and over-the-top (OTT) internet-connected devices via device-specific Hulu applications. Users of Hulu’s digital video service may elect to view content on hulu.com and through external sites such as yahoo.com. To view Hulu content on web, or any Mobile or OTT internet-connected device, Hulu users must subscribe to the Hulu service.
Video Ads are served within the Hulu player at pre- and mid-roll positions. Companion display Ads may also be shown within these placements, or adjacent to the player, in close proximity.
Hulu partners with Content Providers (CPs) and distributors to provide digital video content to users. Hulu also produces certain original video content available through its streaming video service. Regardless of the origin of the content, Hulu maintains control of its streaming digital video players and content pipelines, and encodes all video content and advertisements prior to distribution within Hulu’s proprietary video players on Web and within Hulu applications.
Access to Hulu content is limited to users within the United States, including US territories and military bases overseas, based on IP-address geolocation as determined by a third-party vendor; other regions are blocked.
In most situations, Hulu’s first-party Ad server performs Ad decisioning. However, Hulu also supports third-party Ad serving in certain situations involving pre-established relationships with specified content providers.
The following Hulu metrics are included in MRC-accreditation scope, on the Web, Mobile Web, and Mobile In-App platforms:
- Impressions, which are fired by Hulu clients once a video or rich-media Ad begins playing, or once a display Ad renders.
- Completions, which are fired after a video or rich-media Ad finishes playing.
- Their derivatives – Completion Rate and Delivered Percent.
The following Hulu reports are included in MRC-accreditation scope:
Excluded from Scope
The following items are excluded from MRC-accreditation scope:
- All Hulu metrics not mentioned above in Accreditation Scope.
- All Hulu reports not mentioned above in Accreditation Scope.
- All metrics on the OTT platform, including Impressions and Completions. As of the time of this writing, MRC-accreditation requirements for OTT have not been published.
- Akamai, a content-distribution network used by Hulu.
- Third-party Ad servers that are sometimes involved in Hulu’s Ad serving are excluded. These third-party Ad servers may possess their own MRC accreditations.
- Third-party metrics, including those that measure Hulu-served Ads.
Client-Initiated Beacons Only
All Hulu beacons originate directly from the client. No server-side beacons are used. No proxies are intentionally used.
Hulu does not use sampling. Measurements are counted directly from client-initiated beacons, after filtration processes mentioned below.
Content-Distribution Network (CDN) Usage
Clients send Impressions and Completions to geographically-dispersed CDNs provided by Akamai. Akamai’s CDNs then forward these beacons to Hulu’s measurement servers.
Client initiate Impressions and Completions by firing a beacon via HTTP GET request.
The measurement server returns a 200 to indicate successful reception of the beacon, and writes a line in its log file to record the Impression or Completion.
The above is true of Impressions and Completions for all video, rich-media, and display Ads.
Network latency may result in beacons not arriving at the server around the moment an Impression or Completion occurs, but rather at some point afterward. If this is case, the Impression or Completion is counted upon receipt.
Network issues may also cause beacons being duplicated somewhere along the way (e.g., by a third-party proxy server) between Hulu’s client and server. Hulu addresses this risk in two ways. First, by never performing beacon retries on any accreditation platform (i.e., Web, Mobile Web, or Mobile In-App). Second, by performing beacon de-duplication on the server side.
No Compound Tracking
Hulu does not compound or grouped beacons. Measurements are counted directly from client-initiated beacons one at a time.
Hulu uses a combination of beacon parameters, including a client-generated auto-incrementing “seq” (i.e., sequence) value, to create uniqueness of Impressions and Completions.
Also, in the 200 response to Impression and Completion beacons, the server uses the following Header Controls:
Cache-Control: max-age=0, no-cache, no-store
These values instruct the client to not cache the GET request, and always fire a new instance of the beacon.
Hulu does not engage in auto-play of Video or Interactive Ads. Playback is never initiated without an explicit user action (i.e., click or tap).
Hulu engages in Continuous Play, which is automatic advancement to a new piece of Content after an existing piece of Content comes to an end.
Hulu uses a threshold to determine when to show an “Are you still watching?” confirmation. After four hours of continuous play without user interaction, the dialog is presented, and if no confirmation (i.e.,‘Yes’) is received from the viewer, further playback is stopped.
Based on a preliminary study, ~18% of Impressions on Web was generated by sessions that included 1+ hour inactivity. ~1% of Impressions on Mobile In-App was generated by sessions that included 1+ hour of inactivity. Hulu is developing and conducting improved studies on viewer inactivity in order to update its continuous-play functionality.
Impression and Completion beacons received from Akamai are immediately written into log files. The log files contain raw beacon data. They undergo processing before they can be consumed by reporting mechanisms.
Daily processing allows Hulu’s reports to include the previous day’s data, but not the current day’s.
Hulu’s Mobile application is available to all mobile devices on the platforms mentioned below in the same geographical region as its Web and Living Room offerings. Namely, the U.S. and its territories.
Hulu’s Mobile application runs on iOS and Android.
The use of Hulu’s Mobile application is possible only when the user has proactively downloaded and installed the application. Ad delivery and measurement techniques across application versions have not changed meaningfully in the past several years.
Opened and Initialized Applications
The use of Hulu’s Mobile application occurs only when the user has proactively opened and initialized the application after downloading and installing it. Multiple versions of the same application cannot coexist.
Applications in Use
The use of Hulu’s Mobile application occurs only when the application is in focus. Playback automatically stops when the application loses focus. This means that no Impression or Completion beacons can be fired while the application is background’ed.
Furthermore, during situations when the device goes offline, playback will stop due to the application’s video buffer emptying. At this point, no Impression or Completion beacons can be fired as playback of both content and ads is halted.
While generally no latency issues are particular to OTT devices, there is one exception in that Roku devices have been noted to experience network issues. Further detail is provided in the SSL section below.
Roku devices have been noted to experience network issues. On these devices, it has been observed that having a large queue of beacons to be fired can sometimes result in some queued beacons simply not firing and eventually being discarded inappropriately. Also high volumes of network activity create latency issues and generally slow processing that are particular to these devices.
As a result, as of September 2018, SSL protocol is not available on Roku devices.
Hulu uses a threshold to determine when to show an “Are you still watching?” confirmation. After three hours of Continuous Play without user interaction, the dialog is presented, and if no confirmation (i.e., ‘Yes’) is received from the viewer, further playback is stopped.
Studies are performed on an annual basis, to understand viewer patterns and inform decisions on whether adjustments to improve experience are necessary.
Continuous Play is used more frequently on OTT, as compared to Web and Mobile. As a result, Hulu’s studies on Continuous Play are conducted on a subset of OTT devices.
Detection of TV-off state, and accordingly pausing playback, is entirely dependent on the capabilities of hardware. Where detection is possible, Hulu apps pause playback. However, there is a population of devices that do not communicate the change of hardware state to higher levels in the tech stack. In these cases, Hulu apps are unable pause playback, although Continuous Play thresholds apply as normal.
Hulu has conducted research to gain a general understanding of the aforementioned hardware capabilities and limitations. A study was performed on a broad sample of devices running Hulu’s OTT app. This sample included devices from manufacturers Amazon, Apple, Google, LG, Microsoft, Panasonic, Roku, Samsung, Sony, Tivo, and Vizio. These devices running Hulu were paired with a variety of TVs from manufacturers that included LG, Panasonic, Phillips, Samsung, Sharp, Sony, Toshiba, and Vizio. The study was also ran with and without an AV receiver between the Hulu device and the TV.
A summary of study results is as follows:
- The Consumer Electronics Control (CEC) is a setting on hardware that, when turned on, produces a minor advantage in allowing TV-off state to be communicated upstream.
- Making an input change on the TV usually results in failure of communication of TV-state change.
- Performing a hot unplug of the HDMI cable usually results in success on Roku devices, but failure on other manufacturers’ devices.
- Turning the TV off (i.e., true TV-off) produces mixed results that largely depend on the manufacturer.
- Having an AV receiver between the Hulu device and the TV did not have tangible impacts.
In conclusion, there are many permutations of hardware, settings, and use cases to consider. Improving detection of TV-off is a complex industry issue that will require consistent commitment from TV manufacturers.
Please see the Continuous Play section above.
The methodology for measuring clicks is identical to that of Impression and Completion measurement. A client-side beacon is fired whenever the user performs a click action within Hulu’s application. On the server side, this beacon is logged and undergoes processing, eventually being included in reporting.
Ad-blocking software is capable of preventing Hulu’s Ads from being displayed.
When these situations occur, because no Ads are loaded, no Impression or Completions are fired.
Ad stitching (i.e., using a single manifest with pre-set content and ads for each stream) is performed on the server side, with the result being delivered to Mobile and Living Room client devices to facilitate smooth playback.
Hulu employs a suite of techniques in an attempt to identify and filter invalid activity. These techniques include but are not limited to known and suspected non-human activity and suspected invalid human activity.
Because user identification and intent cannot always be known or discerned by the publisher, advertiser or their respective agents, it is unlikely that all invalid activity can be identified and excluded from report results.
Below, Hulu’s various techniques are summarized strictly at a high-level for security purposes.
General Invalid-Traffic Filtration (GIVT)
Hulu practices GIVT.
Sophisticated Invalid-Traffic Filtration (SIVT) is not performed.
Hulu performs its own filtration without the assistance of any third parties.
Hulu removes all internally-generated traffic.
IAB Blacklist and Whitelist
Based on User Agent, Hulu performs the IAB dual-pass method to remove all traffic that doesn’t appear on the international lists of spiders and bots, and valid browsers.
Activity-Based Filtration (ABF)
Hulu employs ABF to identify anomalies. The specifics of its ABF techniques are kept confidential for security purposes. They are disclosed only to MRC auditors as part of the audit process.
Hulu does not employ any pre-fetching mechanisms. Impression and Completion beacons are fired in real time. This also applies to ads shown as a result of Continuous Play.
Filtration is applied against Hulu’s entire traffic across the Web, Mobile Web, and Mobile In-App platforms. No subsets or sampling techniques are used.
When Impression and Completion beacons are received by the server, they undergo validation to ensure critical parameters aren’t missing. They are also evaluated to see whether parameter values are contradictory. Any beacons that fail either type of validation are discarded ahead of any further processing.
In the unlikely event Hulu’s log files are corrupted, Akamai is capable of resending beacons for Hulu to reprocess.
An analysis of IVT during Q4 of 2016 shows that IVT typically fluctuates at immaterial levels.
When significant changes to Hulu’s filtration methods and / or technology are implemented, Hulu issues a statement to clients with active or upcoming Ad Campaigns. This is reserved for situations where notification provides a discernible benefit to clients. For security purposes, Hulu discloses as little as possible when it comes to its filtration techniques. Disclosures will be accordingly high-level.
Hulu reports a variety of metrics to clients on a per-Campaign basis.
The reports in accreditation scope – Post-Buy and Device – are presented to clients at multiple points in the Campaign, depending on Campaign length and client preference. The Post-Buy is always presented after a Campaign ends.
Estimation is not used. Report metrics are composed solely from actual data collected in the measurement process.
While report data accumulates on a day-by-day basis, it is considered final as, once processed, a day’s data is official and generally unchanging. Data revisions are possible but rare.
These are Impressions and Completions, and their derivatives – Completion Rate and Delivered Percent.
Impressions and Completions are segregated in three ways:
- Platform. This breaks down into Web, Mobile Web, Mobile In-App, and Living Room. The breakdown allows clients to quickly identify MRC-accredited numbers (i.e., Web, Mobile Web, and Mobile In-App).
- Position. This breaks down into pre-roll, mid-roll, and post-roll.
- Gross (i.e., pre-filtration) vs. net (i.e., post-filtration) numbers.]
Non-accredited metrics are also presented. These include, but are not limited to: quartiles, Clicks, Clickable Impressions and Completions, Click-Through Rate (CTR), Time Spent Viewing, and Unique Users.
Open columns with no metrics or data can also be presented. They are meant to reserve space conveniently for manual data entry. These include Estimated Spend values, Third Party and Discrepancy.
Reports can be:
- Run against a specified date range. Typically the entire duration of the Campaign is used.
- Grouped by day, week, or month. Reports use the time zone Eastern Time (ET). A day is defined as the 24-hour period that begins at 12:00 a.m. and ends at 11:59 p.m. ET. A week is defined as the seven-day period that could span Sunday to Saturday, or Monday to Sunday, depending on client preference. Specific dates are provided in reporting for reference.
- Broken down at the Creative (i.e., individual instances of an Ad type) level, or grouped at the Placement (i.e., Ad type) level instead.
Per Hulu’s product options, static companion ads like banners and logo are often displayed alongside video and rich-media ads. These companion ads are segregated in reporting as distinct products.
Campaigns parameters are entered by Hulu’s Campaign Coordinators. Before being set live, every Campaign is subject to full review by a Campaign Manager.
After going live, Campaigns undergo pacing monitoring by Campaign Managers to ensure they’re delivering as expected. Optimizations and debugging are performed if necessary.
Revised metrics, including Impressions and Completions, will be communicated when excluded or invalid activity is detected for a Campaign after initial reporting and within three days after the end of the Campaign. The methods of communication include e-mail statement and / or updated reporting.
IVT Discrepancy Resolution
Campaigns billed against Hulu metrics are never billed for IVT traffic. In other words, Ad Campaign targets are always met after discounting any traffic identified as IVT.
However, as mentioned above, clients are always notified of gross vs. net numbers. After reviewing these, clients may contact their Hulu Campaign Manager to discuss any general questions around IVT volume and classification.
Source data for Hulu’s reports is retained indefinitely without expiry.
Records of data reissuance are also retained indefinitely.
At the time of this writing, Hulu maintains an important relationship with Akamai.
Akamai provides CDN services, playing a pivotal role in beacon gathering, and consequently has a material impact on Hulu ad delivery. Other business relationships exist but are not listed due to having no influence on Hulu’s ad delivery mechanisms.
Per MRC regulations, Hulu has worked with Akamai to complete a qualification process whereby Hulu ascertains Akamai’s stance and progress against anti-IVT measures. While Akamai relies on its clients to make their own IVT policy decisions, Akamai offers a suite of tools to counteract IVT. This suite includes Bot Manager, which allows clients to configure appropriate responses to requests from bots, and use white and black lists. Another tool offered is Kona Site Defender, which provides rate control and analytics, as well as IVT-behavior detection.
Akamai has also granted Hulu access to its SOC 2 security-based report.