Will htaccess pw protection block bots and nerds?
 

https://duckduckgo.com/?q=headless+browser+how+to+defeat+htaccess+paywall &t=fpas

Is old school htaccess with un/pw a bulletproof wall against bots, scrapers, headless browser nerds, etc?

If not, is any method bulletproof?

These days you can 100% mimic a true browser, we do it for a few of the distribution platforms we utilize, they dont allow automated submissions, but we bypass that by using a rendering engine.

I wonder if you might be able to use the 'deny all' line somehow while still allowing true browsers to still access, there must be a way to redirect headless browsers (at least ones that dont mimic being a legit one).

I know next to nothing about .htaccess unfortunately lol

thnx. i'm just trying to find out if htaccess is enough to block bots and scrapers.

guys/gals with paysites...does basic htaccess still work?

and if i make surfers email me for a un/pw...can that process be mimicked by a bot.

These days I think you'd be hard pressed to find a solution that works 100%, back in the day when Ray had his solution (forget the name of it off the top of my head, although I know its on GitHub somewhere for free) its was great but not infallible.

No, htaccess won't be enough for that.

I dealt with this problem last year. In the end, with the help of Chat GPT, I programmed a PHP firewall that works reliably. The firewall blocks all IP addresses that visit more than 80 pages in 5 minutes. Of course, I use a whitelist for known bots such as Googlebot, etc.

Nothing is 100%, but I've been using this solution for almost a year now, and it works well.

:thumbsup

what situation are you looking at? like a paysite, wp site, another cms, etc?

rough plan:


1- landing page. wp. maybe a template like from html5up.net less concerned abt bots hitting this page.

2- 2nd level in. landing page users email to receive an htaccess un/pw to access this free level.

if bots can send and receive email...and use the emailed un/pw......how to defeat that without captcha tech giants involved?

3- 3rd level in is for paid subs.

so two layers of un/pw (2nd and 3rd level) would be required to access.

i usally implement rate limiting into my own custom made apps. for logins, contact submissions, etc.

also for a landing page, i have a free template website, you may benefit from one of the free templates. (mainstream and adult) cams, link directories, blogs, etc. all are free for personal and commercial. no link backs required.

Pretty set on the design and yes, doing link exchanges is good.

was just saying my templates dont require links back ;) just free for anyone to use

i agree, while programming an entire php firewall is an own challenge, you can catch many unwanted bots with simple basic rate limiting on cloudflare on their free tier , this keeps a lot of trash away from your system ressources.

not giving the tech bros my user IP data voluntarily.

HTTP basic auth is vulnerable to brute forcing and the lack of a logging interface can make it hard to audit access.

To answer the question, it really depends on what you're trying to protect. A development site? It's probably enough.

If you're opposed to Cloudflare, have your host setup ModSecurity with OWASP rules. A WAF is pretty much the most important part of this equation, to be honest.

My software, LoginBlue, might be able to help though, depending on your use case. It's a replacement for HTTP basic auth written in PHP that uses two-factor authentication via email to confirm access. It works with Apache and it's tested with Nginx. It runs against an existing user database, so it's a drop-in replacement depending on your CRM or AMS. Paired with a WAF, it's fairly robust. I've had great feedback from clients who use it.

cloudflare, wordfence, crowdsec, fail2ban

Can the script be adjusted? Instead of 80 pages in 5 minutes. 10 clicks/downloads in a minute, etc.

Is it simple to setup?

i create a middleware before the login page is being processed. so you dont have to touch any of the login system, etc.