Industry wide bot intelligence, licensing systematic scraping
The rapid growth of AI has dramatically increased automated access to publisher content. While many organisations scraping your web content are appropriately licensed, others are systematically accessing publisher content without permission or outside existing licensing arrangements.
NLA’s Collective Intelligence Service pools data from participating publishers to quickly identify organisations systematically accessing publisher content. Pooling data has shown we can spot illegitimate use that may otherwise go unnoticed and will lead to new licensing opportunities, protect existing licensed markets and support coordinated enforcement where organisations refuse to engage.
The service is not aiming to replace CDN analytics, bot management or website reporting tools. NLA’s value add is licensing and enforcement capability that uses publisher data, market knowledge and cross-publisher intelligence to identify commercial opportunities and protect the value of your content.
Where organisations are willing to operate legitimately, NLA will seek to educate and engage them bringing them into the appropriate licensing scheme. Where organisations refuse to engage or appear to be operating unlawfully, NLA may recommend coordinated blocking or take collective enforcement action.
The service is provided free of charge to mandating publishers. We expect this to generate additional revenues for you over time.
Frequently Asked Questions
- What is NLA’s Collective Intelligence service?
The service combines web server and CDN access log data from participating publishers to identify organisations systematically accessing publisher content. NLA uses that intelligence to assess whether an organisation is already licensed, represents a potential licensing opportunity, or may require further investigation or coordinated enforcement or blocks.
The value lies in combining cross-publisher data with NLA’s knowledge of the monitoring, intelligence, risk and AI markets.
- Why is NLA offering this service?
Publishers need better intelligence to identify organisations using publisher content outside appropriate licensing arrangements. Without industry-wide intelligence, potential customers may remain undetected and unlicensed. Bad actors may also provide new services that undercut organisations that already pay for content. The service will assist NLA in converting new licensing opportunities on behalf of publishers, protect existing licensed markets and build vital evidence where enforcement may be required.
- What makes this different from other “bot” products?
CDN and bot-management tools are designed primarily to detect and manage automated traffic. NLA is not seeking to replicate those capabilities at this time.
NLA combines cross-publisher data with its market knowledge to:
- identify the organisations behind significant automated access
- determine whether they are already licensed
- understand whether their activities fall within media monitoring, risk, compliance, intelligence, TDM or AI-related markets
- identify potential licensing opportunities
- build evidence for coordinated action where organisations refuse to engage
- gather and share collective intelligence that would not be visible from one website alone
The service complements existing bot management products rather than replacing them.
- How does the service work?
The service analyses publisher web access data to identify organisations systematically accessing publisher content across the network.
The most valuable source of intelligence is full CDN and/or Web Server log data because it includes IP addresses. While many legitimate crawlers identify themselves through user agents, organisations seeking to avoid detection often disguise themselves as ordinary browsers or rotate user agents. IP-level analysis allows NLA to identify distributed activity, correlate behaviour across publishers and build much stronger attribution and evidence.
Publishers do not need to implement new technology or provide data in real time. NLA can work with:
- Secure periodic exports to an agreed location (for example daily or weekly)
- Existing CDN export capabilities
- Direct CDN integrations where appropriate
NLA can support common CDN providers, standard log formats and existing export mechanisms wherever possible.
The objective is to minimise technical effort for participating publishers while maximising NLA’s ability to identify licensing opportunities and support coordinated enforcement where necessary.
- Is publisher data kept confidential?
Yes. Publisher data is handled under strict confidentiality arrangements. Individual publisher information is never shared with other publishers.
Where NLA shares collective findings with participating publishers, information is anonymised and aggregated to avoid revealing commercially sensitive information.
Where full log integration is used, processing is undertaken under a GDPR-compliant data sharing agreement.
- Will NLA block bots on my behalf?
No. Publishers remain responsible for technical controls on their own websites. Where NLA identifies an organisation that appears to require a licence, NLA will seek commercial engagement first. If the organisation refuses to engage, or its activity is clearly illegitimate, NLA may recommend coordinated blocking or take collective enforcement action.
- What information will I receive?
Participating publishers will receive collective updates on:
- Organisations identified as operating across participating publisher sites
- New licensing opportunities being pursued by NLA
- Significant emerging patterns in automated content access
- Organisations under investigation or considered for coordinated action
- Recommended collective responses where appropriate
The service is not intended to provide publisher-level traffic analytics or replace the reporting available from a publisher’s CDN or bot-management provider.
- Can this help identify new licensing opportunities and/or support enforcement?
This is the primary purpose of the service. Where the intelligence identifies organisations making systematic commercial use of publisher content, NLA will assess whether an existing or new licence is appropriate and seek to engage them commercially.
Where an organisation falls within NLA’s mandate, refuses to license and continues accessing content, the intelligence may support coordinated blocking, investigation or legal enforcement.
NLA may also pass relevant opportunities to individual publishers where the appropriate rights sit outside NLA’s mandate.
- What are the benefits of participating?
Mandating publishers will not be charged for this service and could benefit from any or all of the below:
- New licensing opportunities that might otherwise remain challenging to identify
- Increased licensing revenue and future royalty distributions from NLA
- Protect existing licensed businesses and revenue from unlicensed competitors
- Industry wide evidence where organisations refuse to engage commercially
- Coordinated blocking or enforcement where collective action is more effective than publishers acting individually
We expect broad participation – initially with a UK focus – which will give publishers and NLA a clear picture of activity across the market.
For any further questions, please contact [email protected]