Solution to block worldwide services under particular categories

[Aikatsui]

Product: AdGuardHome is a FOSS and On-premises (you own your data) software then i considered to mention resources from other products to get better idea for further development

Why this feature?

1. Definitely that'll be parents/employers most loved feature in this product. It means, you don't have to block steam or tiktok then struggle with mind, what'll my kids use next, they usually find something that parents never have heard for life.
   (ex: there're lot of tiktok alternatives similar chingari, dubsmash, triller, funimate etc)
2. Continuously add new services to 'Block services' page just make it too complicated and when it become large list then you cannot recognize necessary services (or type of services) to block easily
3. You can block worldwide services under certain category. it'll provide most effective coverage
   (ex: block chat apps category then whitelist 'Signal' domains for DNS server users to communicate)
4. 'Blocked services' component suitable for dominant internet companies and adguardhome is a multi-platform software, already has millions of users (ex: 10M+ only docker downloads) then adding regional services leads (ex: social media),
   • to be a one of source that bring foreigners to regional platforms and expand numerous disappointing troubles.
   • to be a advertising/new users generate source to those sites or apps
5. Missing something? You can check query log (or with possible enhanceable mode Collapsed query log for time series data #1164) and report to adguard Report possibility in query log #1133

After checked 3 cloud based providers then below contain appropriate categories(38) list

Parental Control
Enhanced parental control (or adult content filtering)
(It include personals & dating, Intimate apparel & Swimwear and other adult or adult mixed contents)
It seems adguard family dns doesn't block dating sites and similar.

Entertainment
Media Sharing
Streaming Media
Entertainment

Computers
Proxies, VPNs, P2P & anonymizers #101
Online Storage & File sharing
Search Engines & Portals
Visual search
Downloads
Text-Audio only
Content Servers
Torrents

Personal web content & social networking
Chats
Comics, Humor & Jokes
Forums & Newsgroups
Hosted personal pages
Instant messaging
Moderated forums
Blogs & Wikis
Parked sites
Social networking

Games
Gaming
Game playing & Game media
Game-cartoon violence

Illegal Activities/Cheating
Tasteless & Offensive
Illegal/ Piracy software
Weapons

Health & Medical
Fashion & Beauty
Health & Medicine

News
News

Recreation/Hobbies
Food & Dinning
Hobbies & Recreation
Sports

Shopping
Shopping

Travel
Travel

Misc
Real estate & property businesses

Drugs / Alcohol / Tobacco use or abuse
Alcohol & Tobacco
Illegal drugs, use and educations

Resources

Comodo dome shield provide 80+ categories

Adult/Sexual*

• Nudity*
• Pornography*
• Adult contents
• Intimate apparel & Swimwear
• Personals & Dating*

(Note: some words above aren't suitable to add to product in anyway due moral reasons even for adults and children may use/facilitate this software as well)

Arts & Entertainment

• Media Sharing
• Streaming Media
• Entertainment

Business

• Advocacy-NGO
• Auctions & Classifieds
• Business
• Finance & Investments
• Job search & Career development
• Marketing & Merchandising
• Stock Trading

Computers

• Proxies
• Online Storage
• Search Engines & Portals
• Visual search
• Downloads
• Information security
• Online services
• Software-Hardware
• Advertisements & Popups
• Computing & Technology
• Peer-to-Peer
• Text-Audio only
• Content Server

Cultures/Society

• Arts & Society & Culture

Drugs / Alcohol / Tobacco use or abuse

• Alcohol & Tobacco

Illegal drugs/Alcohol/Tobacco use or abuse education

• Illegal drugs

Education

• Education & Reference

Games

• Gaming
• Game playing & Game media
• Game-cartoon violence

Illegal Activities/Cheating

• Tasteless & Offensive
• Illegal software

Health & Medical

• Fashion & Beauty
• Health & Medicine

News

• News

Recreation/Hobbies

• Weapons
• Food & Dinning
• Hobbies & Recreation
• Motor vehicles

Shopping

• Shopping

Personal web content & social networking

• Mobile communication
• Professional networking
• Public Information
• Chat
• Comics & Humor & Jokes
• Forums & Newsgroups
• Hosted personal pages
• Instant messaging
• Web applications
• Kids sites
• Moderated forums
• Blogs & Wikis
• Parked sites
• Social networking

Sports

• Sports

Travel

• Travel

Misc

• Real estate
• Technical Information
• Political issues
• Religion
• Government & Legal

From NextDNS

• Porn
• Gambling
• Dating
• Piracy
• Social network

From Cleanbrowsing

• Adult & Pornography
• Adult mixed contents
• Ads & Tracking
• Torrents
• P2P & File sharing
• Proxy & VPNs
• Gambling
• Dating
• Search engines
• Weapons
• Malicious
• Gaming
• Social networks

[jeee]

Nice. Some additional ideas

• block a category it then maybe be able to enable a certain service (say signal and block all messaging)
• blocking the tiktok category but enable only tiktok for a certain time of day
• give time access for each user (for example 14 year old has access to 30 minutes of tiktok while the 8 year old has none)

[DandelionSprout]

I am of the viewpoint that relying on toggle buttons to block these things are not ideal and also not even properly FOSS-philosophy, since the use of standard filterlists would be preferable and require less effort to check the contents of.

In particular, going to https://filterlists.com/ and searching shalla list, shows raw-domains lists for half or more of the topics listed in the OP.

[neotrak]

When will adguard team add this feature to home? eta?
It's difficult for us to check-block manually. we dont know what'll other person or kid search on internet, sometimes people decide to use sites heard from friends, media sources or find alternatives. For dns server manager, it can be hundred and thousands of online games, social apps and other sites.. i have experienced it myself with adguardhome and i'm not satisfied about blocked services feature, already gave up. therefore we are using paid service provide by a local company for my institute and our home

[ameshkov]

It's not planned on any milestone yet.

The feature request is rather complicated, and the problem is websites' categorization data:

1. It's unclear where to get it.
2. If this data is local, we'll have to come up with an update procedure (which is rather tiresome)
3. Alternatively, we'll have to come up with a web service that will supply this data. Also, not a simple thing to do.

[DandelionSprout]

It keeps surprising me quite a bit that A) a significant amount of AdGuard Home's userbase use AGH for workplace censorship instead of adblocking, and B) that the 18 👍's to the OP ignored #2230 (comment).

To block Flash/HTML5 game sites, Pokémon GO, and a small amount of gacha game resources, these lists should at least be a good start:

• https://raw.githubusercontent.com/cbuijs/shallalist/master/hobby/games-online/domains
• https://raw.githubusercontent.com/cbuijs/shallalist/master/hobby/games-misc/domains
• https://raw.githubusercontent.com/DandelionSprout/adfilt/master/Pok%C3%A9monNoGOZone.txt
• (Optional) https://raw.githubusercontent.com/DandelionSprout/adfilt/master/AntiAnthroCombatWaifuList.txt

To block sites where one can gamble with money:

• https://raw.githubusercontent.com/cbuijs/shallalist/master/gamble/domains
• https://raw.githubusercontent.com/blocklistproject/Lists/master/gambling.txt
• https://raw.githubusercontent.com/Sinfonietta/hostfiles/master/gambling-hosts

[neotrak]

2. If this data is local, we'll have to come up with an update procedure (which is rather tiresome)

This way, users can get updates daily and store in a DB file. comodoshield is slow for us. it shall not ideal unless adguard team run many servers. our local provider work as a middle-man between ISP and customer. they specially do data mining to provide number of services. It's like ADGuard DNS analyze queries history to improve ad-blocking but it's much more less than what that company does. Yeah, it's unclear about a good way to get this data

@DandelionSprout adguard home has censor features. no fever, it does more than what we hope in simple centralized environment and take control over entire network.

[Aikatsui]

@ameshkov It's not planned on any milestone yet. The feature request is rather complicated, and the problem is websites' categorization data: 1. It's unclear where to get it.

See below solutions, maybe you can pick affordable services (one or combine multiple) and some companies interest to provide their services for free or under discounts to foss products but do no integrate them directly, we do not like it then adguard has to do separate task before serve it. These companies have threat and adult content filtering facilities, it's possible to use them for improve adguard default DNS and family DNS

However, I noticed several sites pricing too high then as i said above, it's better to get a service for free or find free services.

• https://zvelo.com/solutions/web-filtering-parental-controls
  Products: zveloDB URL database, Cyber threat intelligence, PhishBlockList | Intelligence, Malicious Intelligence Plus Metadata
  zveloCAT™ - Real-Time Web Content Classification at the Page, Post, or Article Level
• https://www.webroot.com/ie/en/business/threat-intelligence/internet/web-classification-and-reputation-services
  It has two type of services as below,
  Full suite of threat intelligence services for partner integration via SDK and APIs
  Basic URL categorization and reputation data for internal usage via online purchase
  Test site: https://www.brightcloud.com/tools/url-ip-lookup.php
• https://www.cyren.com/ It has web classification service. Products available as for Service providers and Tech companies
  Test site: google.com https://www.cyren.com/security-center/url-category-check
• https://url-classification.io - URL Classification API is a web based API.
  Industry based - parental control, ISP, router application, end point protection, AdTech
• https://website-categorization.whoisxmlapi.com
  https://main.whoisxmlapi.com/ (additionally it has IP geo, DRS, WHOIS and DNS history APIs)
• https://www.netsweeper.com/products/url-filtering-categorization
• https://www.similarweb.com/corp/developer/ Categories and Adult sites APIs
• https://aws.amazon.com/awis
  '
• https://www.webshrinker.com/data-feed/ - Offline category database
• https://url-classification.io - has offline database

[ameshkov]

@Aikatsui ideally, we'd prefer an open-source categorization solution for AdGuard Home integration (if that's possible).

[Aikatsui]

Meantime found that many platforms has this facility since long time then adguard cannot delay to implement easily ;)

Comodo dome shield, Cleanbrowsing, NextDNS, SafeDNS, SmoothWall, FortiGuard, iBoss, Barracuda, Sophos Web Filtering, Titan Content Filtering, OpenDNS, Netnany, FlashStart, DNSFilter, Mimecast Web Security, McAfee Web Gateway, Clouduss web categories, Forcepoint, Codeproof, ScoutDNS, GoGuardian and more
Dnssafety Diladele, NxFilter

I noticed another interesting feature has in many products, real time analyze content, construction, context, URLs then categorize unknown domains. it's possible to build as offline as aggressive feature (otherwise cloud based(ML) solution for all AG products)

@Aikatsui ideally, we'd prefer an open-source categorization solution for AdGuard Home integration (if that's possible).

If there something mention about open-source then found these but it is less likely usable for the purpose as see from outside.

• Untangle Monitor (Open source and GPL). The database is part of WebRoot's Brightcloud, they've have no API option for that directly as it is not their database
  https://www.untangle.com/shop/Web-Monitor/
• Clearos open source filter
  https://www.clearos.com/marketplace/gateway/Content_Filter_Engine

Other resources (Free)

• Sensei in OpnSense
  https://www.sunnyvalley.io/sensei/ (free and commercial)
• Shallalist, use by SquidGuard, Dansguardian (possible with IPCop or IPFire). i dont know if that maintain well now.
  http://www.shallalist.de/categories.html
  http://www.shallalist.de/cgi-bin/stat.cgi (seems not better)
  https://dsi.ut-capitole.fr/blacklists/index_en.php
  http://software.othello.ch/mod_dnsbl/

[OperaART]

Does virustotal db work for this?

[OperaART]

[naDevi]

Please take priority consideration on this feature request @AdguardTeam
Two participants have provided their opinion for shallalist. It's free and offline database. can you guys add it?

https://aws.amazon.com/awis

https://aws.amazon.com/alexa/
Alexa web information service is very cheap. personally i believe they have the best database and you can keep in cache every queried domain category some years before re-update. this is a nice feature for all adguard products (@Aikatsui 🌟)
1,001 to 1,000,000 requests/ month - $0.00045 per request
Over 1,000,000 requests/ month - $0.0003 per request

Edit:

• get 1 million domain list according to popularity (rank) eg: from google to below (the problem, not sure if possible)
  • check the category (maybe it can get at same time?)

It's better to dicuss with amazon tech support and make clear everything before add funds otherwise $450 will be evaporate within second 😅

[ameshkov]

Interesting, thank you for the info!

Meantime, I'll see if we can use Shalla List.

[DandelionSprout]

Shalla normally has licences that discourage use by companies (https://www.shallalist.de/), and it's not 100% obvious if the Shalla team is aware of the GitHub re-uploads of their lists (https://github.com/cbuijs/shallalist/).

If that was to be sorted out, I see no problems with using at least some of Shalla's lists.

[ameshkov]

I've contacted them to ask if it's possible to use it in AGH

[naDevi]

@ameshkov Any news?

[ameshkov]

Surprisingly, no response yet.

[naDevi]

Surprisingly, no response yet.

I called to the phone number available on this page (2 days) http://www.shalla.de/impressum.html but they didn't answer.

[Aikatsui]

I noticed mentioned one list offer under creative common license https://dsi.ut-capitole.fr/blacklists/index_en.php recently
The maintainer "fabrice prigent" interest about adguardhome, may be already following this project

@ameshkov It has around 60 categories, update daily and the list maintain since 25yrs. It's possible to download via http, ftp, rsync
If it good then i suggest to assign this FR to milestone because upto 20 competitors have this feature already.

[ameshkov]

Thank you for the suggestion! I am still not ready to assign a milestone, though, I'd only be able to do that after the redesign is completed.

[piyushgarg]

Either categorize them or put the labels like (chatting, video streaming, gaming, betting, porn, shopping, photo sharing etc) this on the services.