Portal   Forum   Members   Market   Gallery   Events

robots.txt

Discussion in 'Feedback' started by KroeT, May 15, 2002.

  1. KroeT

    KroeT Peasant

    Joined:
    Jun 7, 2001
    Messages:
    659
    Likes Received:
    0
    Market Rating:
    0
    Jake, do you have a robots.txt somewhere in your root to prevent webcrawlers from indexing your forums?

    If so, what does it say?


    I heard something like this, and would like to have it on my forums as well cos well, ehm, a webcrawler came by last month and pumped an extra 20GB on my traffic. Me no like. :D
     
  2. Jakeman

    Jakeman MSC Founder and Donator

    Joined:
    Feb 26, 2000
    Messages:
    25,756
    Likes Received:
    27
    Market Rating:
    16
    nope. should i? :retard:
     
  3. KroeT

    KroeT Peasant

    Joined:
    Jun 7, 2001
    Messages:
    659
    Likes Received:
    0
    Market Rating:
    0
    Hmz, someone told me I should, and your forums are bigger than mine (me = puny).

    I'll look around, it was something like

    # for all robots
    User-agent: *
    # indicate that no robots should visit this site further
    Disallow: /


    in a file called robots.txt. Could be BS, but I'm not sure.
     
  4. cowofwar

    cowofwar Peasant

    Joined:
    Jun 14, 2000
    Messages:
    13,721
    Likes Received:
    18
    Market Rating:
    0
    Yes. Stick a robots.txt in there. You don't want those damn spiders in your forum.
     
  5. Jakeman

    Jakeman MSC Founder and Donator

    Joined:
    Feb 26, 2000
    Messages:
    25,756
    Likes Received:
    27
    Market Rating:
    16
    what goes in the file? what text?
     
  6. cowofwar

    cowofwar Peasant

    Joined:
    Jun 14, 2000
    Messages:
    13,721
    Likes Received:
    18
    Market Rating:
    0
    Mine.

    Code:
    User-agent: *
    Disallow: admin.php
    Disallow: /admin/
    Disallow: /images/
    Disallow: /includes/
    Disallow: /themes/
    Disallow: /blocks/
    Disallow: /manual/
    Disallow: /modules/
    Disallow: /language/
    
     
  7. KroeT

    KroeT Peasant

    Joined:
    Jun 7, 2001
    Messages:
    659
    Likes Received:
    0
    Market Rating:
    0
    oooooh, I said something useful.

    :eek:
     

Hitometer: 53,555,575 since 1995