how to find the duplicate urls within tousands

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • zentz
    Confirmed User
    • Nov 2003
    • 8062

    #1

    how to find the duplicate urls within tousands

    i have a .txt file with tousands of urls. now my script has reported that there are 20 duplicate urls but it cant tell which ones are they. is there a script or a software that can find and list the duplicates for me so i can fix them properly ?
    Programs that owe me money ---- Epassporte.com ~ $2700 | Protraffic.com ~ $2600 | XonDemand.com ~ $3000

    Email: [email protected]
  • pussyluver
    Clueless OleMan
    • Mar 2003
    • 11009

    #2
    Use the search function in notepad. Start at the top.

    Study Visual Basic and write a program.


    There's a couple of ideas. Hope someone has something a bit more direct.

    Comment

    • latinasojourn
      Confirmed User
      • Oct 2003
      • 3191

      #3
      Links Suite 4

      Comment

      • pudcat
        Confirmed User
        • Mar 2003
        • 1169

        #4
        if you know php or some other scripting language then it's pretty damn easy.

        One method:

        split it into an array
        loop through the array
        check if you've already seen that url, if not then remember it
        print out a list of urls that you've seen

        probbaly a cleaner way though I'm lazy
        SUBMIT YOUR BABE GALLERIES

        PROMOTE YOUR BLOG HERE

        always looking for hardlinks icq #207011694

        Thunder-Ball.net, good for hardlink exchanges

        Comment

        • spunkmaster
          Confirmed User
          • Jan 2004
          • 2052

          #5
          Go to download.com and get one of the super free notepad programs
          and do a search and you'll find them really fast.

          Comment

          • darksoul
            Confirmed User
            • Apr 2002
            • 4997

            #6
            From shell:
            Code:
            sort file.txt|uniq -d
            1337 5y54|)m1n: 157717888
            BM-2cUBw4B2fgiYAfjkE7JvWaJMiUXD96n9tN
            Cambooth

            Comment

            Working...