[ nemnesic @ 06.03.2013. 12:02 ] @
Jel moze nego da mi napise Rewrite rule da blockiram facebookexternalhit bota?

Inace problem je u tome sto Facebook bot ima neku bug i u poslednja 3 sata nam flood-uje webserver i CPU se spike-uje do 90%.
Pa hocu dok oni ne srede taj bug da blockiram a sam ne umem a nas server admin spava (zbog vremenske razlike...nije lenj).

evo dela is access_log-a

Code:

10.244.142.212 - - [06/Mar/2013:05:55:14 -0500] "GET /image/renderProductImage?id=37 HTTP/1.1" 503 409 "-" "facebookexternalhit/1.1 (+[url=http://www.facebook.com/externalhit_uatext.php)]http://www.facebook.com/externalhit_uatext.php)[/url]"
10.244.142.212 - - [06/Mar/2013:05:55:15 -0500] "GET /image/renderProductImage?id=37 HTTP/1.1" 503 409 "-" "facebookexternalhit/1.1 (+[url=http://www.facebook.com/externalhit_uatext.php)]http://www.facebook.com/externalhit_uatext.php)[/url]"
10.244.142.212 - - [06/Mar/2013:05:55:15 -0500] "GET /image/renderProductImage?id=37 HTTP/1.1" 503 409 "-" "facebookexternalhit/1.1 (+[url=http://www.facebook.com/externalhit_uatext.php)]http://www.facebook.com/externalhit_uatext.php)[/url]"
10.244.142.212 - - [06/Mar/2013:05:55:15 -0500] "GET /image/renderProductImage?id=37 HTTP/1.1" 503 409 "-" "facebookexternalhit/1.1 (+[url=http://www.facebook.com/externalhit_uatext.php)]http://www.facebook.com/externalhit_uatext.php)[/url]"
10.244.142.212 - - [06/Mar/2013:05:55:15 -0500] "GET /image/renderProductImage?id=37 HTTP/1.1" 503 409 "-" "facebookexternalhit/1.1 (+[url=http://www.facebook.com/externalhit_uatext.php)]http://www.facebook.com/externalhit_uatext.php)[/url]"
10.244.142.212 - - [06/Mar/2013:05:55:15 -0500] "GET /image/renderProductImage?id=37 HTTP/1.1" 503 409 "-" "facebookexternalhit/1.1 (+[url=http://www.facebook.com/externalhit_uatext.php)]http://www.facebook.com/externalhit_uatext.php)[/url]"
10.244.142.212 - - [06/Mar/2013:05:55:16 -0500] "GET /image/renderProductImage?id=37 HTTP/1.1" 503 409 "-" "facebookexternalhit/1.1 (+[url=http://www.facebook.com/externalhit_uatext.php)]http://www.facebook.com/externalhit_uatext.php)[/url]"
10.244.142.212 - - [06/Mar/2013:05:55:16 -0500] "GET /image/renderProductImage?id=37 HTTP/1.1" 503 409 "-" "facebookexternalhit/1.1 (+[url=http://www.facebook.com/externalhit_uatext.php)]http://www.facebook.com/externalhit_uatext.php)[/url]"
10.244.142.212 - - [06/Mar/2013:05:55:16 -0500] "GET /image/renderProductImage?id=37 HTTP/1.1" 503 409 "-" "facebookexternalhit/1.1 (+[url=http://www.facebook.com/externalhit_uatext.php)]http://www.facebook.com/externalhit_uatext.php)[/url]"
10.244.142.212 - - [06/Mar/2013:05:55:16 -0500] "GET /image/renderProductImage?id=37 HTTP/1.1" 503 409 "-" "facebookexternalhit/1.1 (+[url=http://www.facebook.com/externalhit_uatext.php)]http://www.facebook.com/externalhit_uatext.php)[/url]"
[ cyBerManIA @ 06.03.2013. 12:45 ] @
Da se ne majes agentima, dodaj ovo u .htaccess

Code:
order allow,deny
deny from 10.244.142.212
allow from all 
[ nemnesic @ 06.03.2013. 13:09 ] @
Citat:
cyBerManIA: Da se ne majes agentima, dodaj ovo u .htaccess

Code:
order allow,deny
deny from 10.244.142.212
allow from all 


hvala na odgovoru ali na zalost ne mogu da menjam .htaccess
moram da menjam vhosts.conf
[ nkrgovic @ 06.03.2013. 13:53 ] @
Citat:
cyBerManIA: Da se ne majes agentima, dodaj ovo u .htaccess

Code:
order allow,deny
deny from 10.244.142.212
allow from all 

Ne radi ovo!

Ovo 10.* je neki interni IP, verovatno imas neki load balanser / reverzni proxy, koristis amazon aws ili tako nesto... Ako ovo blokiras, blokiraces ceo sajt svima.

Vidi mozes li da dobijes X-Forwarded-For header, pa po njemu da radis.
[ nemnesic @ 06.03.2013. 13:57 ] @
upravo tako 10.244.142.212 je load balancer. i Jeste ovo je na AWS.

probao sam nesto ovako ali ne radi. svi dobiju 403. ne samo facebook.

Code:

RewriteCond %{HTTP_USER_AGENT} ^facebookexternalhit/1\.1\ \(\+http://www\.facebook\.com/externalhit_uatext.php\)$ [NC,OR]
RewriteRule ^  -   [F]
[ nemnesic @ 06.03.2013. 15:08 ] @
pa da li je moguce da niko nece da mi ovo na brzinu odradi?
[ Tyler Durden @ 06.03.2013. 15:22 ] @
Jesi probao ovako?

Code:

RewriteCond %{HTTP_REFERER} ^.*facebook\.com*.$ [NC]
RewriteRule .* - [F]


U svakom slučaju najbolje bi bilo da to blokiraš prije http servisa na nivou firewall-a.
[ nkrgovic @ 06.03.2013. 17:33 ] @
- Dodaj u apache logove XFF i loguj ga.
- Vidi sa kog IP-a ti dolaze ti zahtevi
- Dodaj u firewall na amazonu te IP-ove i blokiraj ih.
[ cyBerManIA @ 06.03.2013. 19:22 ] @
LOL nisam ni overio brojke, ja sam po defaultu copy/paste, kad ono 10.0.0.0/8
Uvek duplo proveri kad neko nesto napise.

---
Ok, ipak da se vratimo na user agent, ipak ga fb nece menjati svaki cas.
Pretpostavljam da imas Apache i mod mod_setenvif i

SetEnvIf User-Agent ^facebookexternalhit/1\.1 zabrani
<Directory /direktorijum>
Order Allow,Deny
Allow from all
Deny from env=zabrani
</Directory>

gde je /direktorijum unesi dir do www, najverovatnije /usr/www; /www i slicno.

---
Ako ne uspe sa useragentom, blokiraj ih na firewallu.