It makes a lot more sense to ad block system wide than to expect every bit of software you use to implement extensions to allow you to do it. There are plenty of services like AdGuard and Lockdown, they can use the same lists that ublock uses, and you protect not just your browser but your email and everything else you use. Some routers will let you do something similar, and you can add a pihole to your network if your router won’t help.
It would be great to move adblocking to an OS/network level, but DNS blocking just isn't comparable to what a browser-level adblocker does.
To get to the point where I would feel comfortable having my OS handle browser blocking, a reasonable chunk web browser functionality would need to be moved out of the browser and onto the OS. I just don't see that happening any time soon, and I'm not sure that's the direction we would want to move with browsers anyway.
Ublock Origin will do things like stub Javascript methods on the page. A Piihole can't do that. And even for simple things like blocking requests based on the current domain -- the movement towards DoH and SNI are going to make that harder and harder as time progresses. You really need an interface that has insight into not just what requests you're making, but where/why you're making them.
Not to say that system wide blockers don't have value, but they're really there for the apps that can't handle their own adblocking. They're a defense in depth for the requests that slip through other parts of your setup, but they're not a good replacement for a browser extension.
All you're asking is already possible with Privoxy[1], which is even stronger than a browser adblocker. It's a very old software: it used to be unmaintained and lacking some essential features, but thankfully the development resumed and is now fully fuctional again with the modern web.
It can be used as an adblocker based on domain, request path, HTTP headers, etc, but it can do much more. It can redirect requests (for example, replacing assets from a CDN with a local cache), modify headers (stripping or making cookies temporary, changing user agent, etc.) and even rewrite the content of web pages using regular expressions or any external program.
By default, it has only a basic configuration that blocks tracking and ads, but there are tools[2] that convert adblock rules to the Privoxy format, so it will be functionally equivalent to adblock.
It acts as a CONNECT proxy, so you can run it locally or on a router and if combined with a NAT rule, it can also work transparently (obviously, you need to manually trust a CA certificate for https).
Can it let me pick any visible HTML element on a website to filter out just by clicking on it? Can it block content of a website without interfering with a `curl` request or a file download from a random messenger app?
All those things are trivial to do in a browser plugin but probably a total workaround-filled pain on any other layer in the system.
> Can it let me pick any visible HTML element on a website to filter out just by clicking on it?
The best it can offer is a CGI editor to change its configuration from the browser. I don't see how you could implement something like this: it's either interactive or a passive network element, but not both.
> Can it block content of a website without interfering with a `curl` request or a file download from a random messenger app?
This is really trivial, just don't proxy them (ie don't set the http_proxy variable).
Oh, Privoxy. Must've been 20 years since I used that. But, can it block certain elements with certain IDs? And do I have to disable DoH to make it work?
If you mean hiding an element, yes it's possible: you can either inject CSS into the page or write a filter to remove the HTML entirely. For example, adblock2privoxy generates both Privoxy rules to block requests and stylesheets to hide elements (you need a local webserver for this, though).
> And do I have to disable DoH to make it work?
It's Privoxy, not the browser, that will do the DNS queries. So, no: it will work regardless of DoH.
1) The converter you link has 62 stars and hasn't been updated in 2 years. Additionally I'm seeing multiple issues about basic adblock rules not taking effect. Short version, I would not trust this repo to convert rules.
This is kind of exactly what I'm talking about with the difficulty of keeping pace with what is essentially a shared standard in the adblocking community. It's not enough to write one converter that gets updated every 2 years, in the space of those 2 years, Ublock Origin has expanded the syntax it supports. Adblocking is a cat-and-mouse game, there isn't a single set of features that can implemented once and then the software marked as "done".
2) Even assuming that converter does work (which I am doubtful of), Ublock Origin uses a superset of the adblock rules format, so you have to target what Ublock Origin supports, not just what adblockers in general do.
And obviously I'm not going to try and recreate those lists myself manually, I don't have the time or energy to do that. They have to be 100% consumable from upstream.
----
Okay, moving on to Privoxy itself:
1) On the community aspect again, I don't see dedicated Reddit groups devoted to finding every single broken website on this software. I don't see a public issue tracker. It seems to be following the old FOSS philosophy of developing software primarily on mailing lists someplace, which is fine for some software but not fine for something that is highly community dependent like adblocking.
You say the software is being actively developed again, I don't see any way to easily confirm that. I don't see any way to easily figure out how many people are using this and verifying that it works.
2) As far as I can tell, this doesn't support DoH. That is also kind of a dealbreaker for me, I don't want to make myself less secure in one area to make myself more secure in another. This is a solveable problem: if Privoxy was being set up as a local DoH server as well, and it was using DoH itself to query/cache results, then the issue would almost completely go away.
However, am I correct in guessing that Privoxy is also going to struggle in the future with encrypted SNI, or with the fact that my browser strips referrer headers from requests?
3) I'm looking at Privoxy's pattern documentation[0], and correct me if I'm wrong but it doesn't seem to support contextual blocking at all. In Ublock Origin I can do rules like:
$script,third-party,domain=imgbox.com
My original criticism of DNS blocking in general was that it lacked context information, so it's just flat-out not acceptable for a Ublock Origin replacement to lack the ability to distinguish between a third-party request and a first-party request. That's critical functionality. Maybe I'm missing something here, but I've gone over the Actions and Template file documentation and I don't see the words "third-party" even mentioned anywhere.
4) Privoxy seems to lack the ability to block iframes, or at most it seems to have the ability to strip them from the HTML itself. That's not enough, sometimes iframes get dynamically created after a page is loaded, and modifying the HTML is not enough to block that.
5) I don't see any way to mark sites as trusted (probably related to point #3). So there doesn't seem to be a way for me to disable Privoxy when I'm on a specific site.
6) I don't see anything in the docs about CNAME unmasking. And CNAME cloaking isn't a theoretical attack, there are websites in the wild using that technique.
7) Browser integration also seems to be lacking. This isn't the biggest problem, I can tolerate annoyance, but it's a little bit of a quality of life issue.
8) And so on. Most of UBlock Origin's dynamic filtering syntax[1] seems to be unsupported. It's very possible I'm misreading the docs, or the docs are out of date or there's a trick to make it work, but if that's the case, that's also a problem, because then the docs need to be clearer.
----
None of that is to say that Privoxy is bad software. It's just to say that it doesn't seem like it's an all-in-one replacement for what my browser does.
We often do defense-in-depth in this area. You can set up a Piihole, or a firewall, or a proxy server to handle adblocking for devices and applications that don't expose interfaces like the browser does. That's a good idea. But the farther away from the context of the application that you get, the harder it is to do really detailed blocking based on that context.
This is something fundamental about adblocking that people don't always seem to understand -- it's not an either/or proposition, it's not like you set up a proxy server and all of your browser configs become useless. The proxy server just adds another layer of defense.
No, there's nothing javascript specific in Privoxy, but there's also nothing stopping you from implementing it. You could write filters to inject custom code (like greasemonkey userscripts) or modifying the scripts in tranport.
It makes sense to use both, but there is no alternative to "context-aware" ad blocking within the browser. YouTube video ads cannot be blocked via hosts file, same for any ad that's served directly from that site.
Also uBlock Origin lets you block any HTML element on any site. This lets you (interactively!) cut out autoplaying videos, annoying popups and basically anything that you as an individual user find distracting.
Or how about the option to block loading of large images? Try implementing that system wide without accidentally breaking certain requests using `curl` and while still allowing the user to flip that switch in seconds.
There's no single abstraction layer where it makes sense to do everything at once.
Internet Explorer has less than a 2% market share in the US, so under normal circumstances (given the sacrifices you’d have to make everywhere else) not bothering to support is is often a sensible decision. When it’s an access to healthcare issue though you can’t really exclude 2% of users.
That said, building something that works well on mobile (re your point b) but also works on internet explorer is a pretty tough ask. No reason you couldn’t build a basic bitches version of the site that works on old devices, then route traffic these based on user agent.
Idk, points a through c are basically already solved problems. I’d be disappointed to see problems like that if I was spending $20,000 let alone millions. Issue d though does feel like the sort of thing that you didn’t know was on the horizon up front, would be a pretty major rebuild to add in later on.
> No reason you couldn’t build a basic bitches version of the site that works on old devices, then route traffic these based on user agent.
Why isn’t this just the site?
I love Spartan, ugly, functional sites. That’s what the government should build, always. They should have style guidelines that are basically “no style”.
You’re not competing with anyone, there’s no need to look good. Function, function, function.
One reason that comes to mind is that a lot of people browse on mobile devices. These days a site really ought to be responsive to mobile viewports and a lot of “basic bitch” government sites are not.
Plain old HTML stops being responsive as soon as you try to do anything more than text. A plain HTML table would be horrible on mobile. If you want to reflow anything that isn't text you will need flexbox. There is literally no reason to not use CSS to make a site that works well on mobile and desktop.
I’m not convinced that’s true. Case in point: tabular data. I don’t think presenting something in a table is an attempt to make it look good, I think it’s the most logical and recognisable way of presenting certain kinds of information. But tables don’t scale well on a mobile viewport without extra work.
Part of function is layout. It isn't any more difficult to grab Bootstrap or some other out of the box framework that'll give a good consistent look than it is to go "no style".
There can be a medium. US Gov sites generally look awful. The UK Gov Digital team have done a way better just of standardizing things in ways that look nice.
>>Internet Explorer has less than a 2% market share in the US, so under normal circumstances (given the sacrifices you’d have to make everywhere else) not bothering to support is is often a sensible decision. When it’s an access to healthcare issue though you can’t really exclude 2% of users.<<<
That thinking is exactly the reason why this system turned out to be problematic. Even if it wasn't for healthcare purposes, bottom line is - you have to know who your 'actual users' are. Elderly people (60s and above) are still heavy IE users and so are quite a few government offices. If these 2% of overall browser users are the majority of your users or your early users, then not supporting them becomes a problem.
Detect IE, redirect to a static site with a 1-800 number, have a call centre staffed with people running Chromebooks schedule an appointment over the phone.
Probably cheaper overall, and certainly a better user experience for the tech-challenged seniors.
Something to note about this strategy is that older versions of IE (10 and older) only support TLS 1.0 by default. Below 8 you're stuck with TLS 1.0 and can't even enable anything higher. So if "https://www.cdc.gov" is using modern web security IE users won't even get to request data from the server, so you can't detect their browser.
To add more info. TLS 1.0 is prohibited from usage by many regulations and it will be immediately flagged by any security audit/tools. The government contract most certainly includes clauses about security and following recommended practices.
In short, anything below IE 11 is de-facto out of the picture.
> But the GOV.UK pages are written in simple HTML. They are designed to be lightweight and will work even on rubbish browsers. They have to. This is for everyone.
2% of traffic doesn't mean that 2% of users only have access to IE6. It may limit which computing device they can use but not supporting IE6 isn't blocking millions from your site, it is blocking them from using your site on all of their devices.
Probably not true. It means blocking them from using the site because they only have 1 or 2 devices and those are running IE.
The article talks about the fact that a lot of the seniors get help with their online stuff from the libraries (which are currently closed). What is the probability that folks like these have multiple devices at home and/or they can use them?
Years ago, a lot of my bosses at work (in Tech) still used IE. Lots of our clients used government issued computers which were locked down to IE, sometimes several versions behind.
At the end of the day, it boils down to - know who the actual users are.
never mind all the accessibility rules you have to follow (keyboard navigable, alt text for screenreaders for images, minimum contrast for sight impaired, constrained color scheme for colorblind, etc). . . that greenfield SaaS app made by 5 22 year olds isn't going to cut it here.
> Internet Explorer has less than a 2% market share in the US
This is true but if you are a Government agency in a Country of 300M+ people, that 2% is still several hundred thousands users, who are calling your tech support etc. etc.
If this was a private app then it would be different, IMHO Government websites have accessibility concerns that the rest of us can ignore.
I opened a parler account to see what it’s like. Everything suggested to me was right wing pseudo news sites and pundits along with republican lawmakers. Aside from a small amount of gaming content I couldn’t find anything vaguely interesting.
Isn’t that the way laws work? You can’t prosecute someone until they’ve committed a crime. The issue here is whether the “crime” is tweeting stuff that starts an insurrection, or tweeting stuff that could be interpreted as potentially starting an insurrection.
Me telling someone to go fuck themselves probably isn’t a problem, but me telling someone with learning difficulties to go fuck themselves and them feeling compelled to do it and hurting themselves probably is a problem.
