Is Anyone Using The Full Data Dump From HubTrafiic?..
 

What tools are you using? - It's massive and I am struggling...

Thanks.....

You should be careful with word choice otherwise you can attract currently sober into thread :D
And yes i use full dump as well, i made my own script to parse it plus script to parse weekly updates.

Good luck, I have never been able to successfully full from those dumps. Hope someone can shed more light on it as well.

lol - True...

yes I have the full pornhub/redtube/tube8 dumps on my sites

like klentelaris I use my own script

you can fit the whole thing on a single elasticsearch node but you need a decently beefy machine

it's definitely too big for most of the cookie cutter scripts people use, smart-cj, wp-tube, tube-ace

I bet mechbunny could handle it though, with a big enough server

I haven't looked at ElasticSearch yet - Is easy enough to import it?..

I do know those script are extremely resource hungry for some reason, tho when i was testing with my script it worked fast on a 512 mb vps.

those script are extremely resource hungry they all are we need some way to break it down above cuts off at set amounts then carries on very slow

:1orglaugh:1orglaugh:1orglaugh:1orglaugh nope!

LOL I checked out some of their demo tubes. Barely functional, slow as shit, some of them are already shut down

good fuckin luck with that

telll me about it, mines stuck on php5 lol if i upgrade anything i think it will break

next.......... lol

i think i will have to start from scratch and go with the guy who is jetting around the world and airport staff are smashing his equipment what is his tube script?

One way to deal with it is to import it to a No SQL database and use something like elastic and an api to feed a front end site and give search functionality. This is all free, you just need to know how to put the pieces together.

The advantage of doing it this way is you can run just one box to feed an unlimited number of websites with data through the API.

That's pretty much what I have been doing most of the day - There has been a lot of head scratching and swearing but I think I am getting there...

Thanks.....

Just a tip - SSD drives make a big difference in performance.

You are so comical lately . "only way to deal with it" :1orglaugh
And speaking about databases, i tested mariadb while i was doing import of this for first time, but for some reason , while mariadb loads content faster then mysql, it is terribly slow when it comes to inserting-it took 12 hours to import hubtraffic dump, while with mysql it took only 3 hours. So i figured out how to optimize mysql to load content fast and there was no need anymore for mariadb.

one thing this all reminds me of when tube scripts and xml/atom/css like some of you i was trying different ways to get this to work,

that i fucked up score feeds somehow and started showing hundreds of thousands of hits in the program admin, he was asking what the hells going on, i will see if i can take some screenshots, he didnt close account as i generally had no idea what was going on, still dont

You need to learn to read, I said "ONE WAY TO DEAL WITH IT" :helpme

no joy can only go backstats 2009, this was 06/08

Damn brain auto-correct :1orglaugh

https://i.imgur.com/rItRjGt.jpg

:thumbsup:thumbsup:thumbsup

maybe find a tool that can split it to smaller files

I wrote a python script to deal with it, keeps everything in memory (I have an old server with tons of ram) and it finishes in a few minutes.

what did you name it?

white snake???? :1orglaugh:1orglaugh:1orglaugh:1orglaugh:1orglaugh :1orglaugh

split axe may do it. i have a copy if u need it

for huge files i have used "load data infile" works best if you dont want any manipulation done on data.

I just found/used a prog from filesplitter.org - It's Freeware and worked like a charm - Not bad, it's a 9GB file and the prog looks like it was made for Windows 3.11...

:thumbsup:thumbsup glad you found one! cheers!

On any unix/linux/mac system you can use the command split to split files.

example splits a file of any size into smaller files, each with 5000 lines

If split isn't on your system:

on Ubuntu/Debian type

on Centos/RHEL

on MacOS

Ooops - I split into 500mb chunks and I am doing a find and replace | to ^t - I probably should gone a bit smaller - lol...

How did you go, did you get there?

how did that go for you 9gb to a 500mb then what? did it work?

No I just use their API to pull in the good videos I want and bypass all the junk shitty vids.

The API is good, but if you're building out a massive embed site the dump is better.

Too easy. Although I am using the API for some stuff like tags etc...

Too be honest it's a chance to learn about 'big data' and related tech. I can then use that knowledge for something that might actually make me some money!....

Good idea, learning stuff like distributed database, elastic etc is all useful.

If you're keen to learn the most popular PHP framework around, which includes support for technologies related to big data, such as Algolia then learn Laravel.

https://laravel.com

https://laracasts.com

I have been thinking about Laravel - I looked at it years ago - Maybe it's time to look again...

OK - So this is the flow I used:

Download/Unzip CSV dump -> Use FileSplitter to split CSV into 500mb files -> Used NotePad++ to find and replace pipes with tabs (makes DB import easier) -> Used Navicat to import into MongoDB...

Now I just have to learn how to use the data lol.....

Notes: This was all done on a local Windows machine - Navicat will import them into an external MongoDB instance if needed - It's a lot of data and you need to be patient - Either that or start a large Google Cloud instance and do it on there, I wish I had done that lol...

people using API ,

API Limit is about 40 requests per 10 second

so its not feasible to use api for search queries/tag queries on live site, you need to implement your own search function.

it's fine if you cache requests.