Piano World Home Page Forums Our Most Popular Forums Adult Beginners Forum OT: Spiders (NOT the 8-legged ones!)

Just wondered who/what the 'Spiders' are?

Who's Online?
188 registered (acortot, 4evrBeginR, -Frycek, 8ude, Acoustic Dave, Agilita), 823 Guests and 86 Spiders online.

Edit: unless the arachnids really HAVE gained sentience and are populating the PW forums...?!

I once found a wierd half-done web site that listed those of us in the recitals and what pieces we played, as part of a larger database I think, where you could supposedly look up a song and get a list of on-line places to hear it. Something like that. It was nebulous to me.

Anyway, someone suggested that the database was put together by an automated program that combs the web for information - he said he wouldn't be surprised given the number of spiders here. So I associate them with data-finding programs.

I'm sure the techies here will know more.

Cathy

Spiders (or spyders or web crawlers or a bunch of other names) are automated programs from places like Google that "crawl" around the web following link to link and indexing all of the pages they can find for the search databases of the world. Some spiders are looking for web content to index (like Google) but others are just looking for email addresses or other specialized information.

In other words, a spider is a program "browsing" the web rather than a person.

It's impressive how many of them are browsing us at any given moment!

Frank assured us once that spiders are a Good Thing. They help insure that PW comes up near the top in any google search. That helps Frank attract advertisers. And that helps keep PW afloat.

Spiders are the way to get your web content added to the major search systems. But they are also the way people give away their email addresses and other personal information in signature blocks, personal web pages, blogs, and similar ways.

There is no alternative today if we want the web to be broadly indexed and searchable, but folks just need to know that anything they put on public sites like this, or anything they put on personal sites that are linked to public sites (through signature blocks for instance) are fair targets to any spider anywhere.

Yup! And, not only does PW come up in the top of Google, Google indexes posts here SUPER FAST.

People go through all kinds of pain trying to "get indexed" (have Google recognize their site and start showing it in search results).

I've noticed - if I post something here, then do a search for that topic on Google, my post shows up (near the top of the results even!) really strangely fast.

I think it means that Google likes us here probably because there's so much going on, lots of new stuff added daily. Or maybe we're just a likeable crowd

At any rate ,we're being watched!

Though the idea that spiders are watching us is strangely disturbing to an arachnophobe like me

Hi Currawong! Not to worry... I just learned tonight, according to my piano teacher... spiders do not like the smell of sheep (thus, sleeping on wool will keep spiders at bay.)

I'm not sure how we end up on this stuff sometimes

So the remedy here should be very easy Wool blankets for everyone!!!

Ah, some piano teachers are just full of really useful information! Thanks for that

Can spiders see inside a member's profile? I've noticed that some members have their email there but not in their signature.

I assume that spiders can't read text that is part of a graphic image, so on my website I made a small .gif image that shows my email address. Humans can read it but they have to type it instead of clicking on it.

In general, spiders don't analyze images nor can they find web resources hiding behind logins and passwords. There is no technical limitation that would prevent someone from pairing a web crawler with some image analysis software, but most spiders today still deal only with text harvesting. It's easier and suits most purposes.

One of the other things you see sometimes is email addresses that have been "disguised" to fool programs looking for the @ sign syntax that distinguishes an address but that still try to be obvious enough for a human being to decode: "smith(at)xyz(dot)com" might be an example.

Some researchers have called the wealth of web stuff hiding behind commercial login screens the "hidden internet" and estimate it may be as extensive as the visible internet (although there is really no way to know). I'm taking a couple of grad courses this semester and the university has given me subsidized access to hundreds of online resources that I couldn't otherwise access without buying a personal password.

The scope of this hidden internet is really fascinating: virtually all scholarly literature published in the last 15-20 years is available online in full-text PDF and there are archives in some subject areas that go back much further than that (I just downloaded an article published in 1962 in a fairly obscure information science journal). Little of this kind of content is available to spiders.

Thank you Packa. Very helpful information.

Spiders are those heavy contraptions with large casters that grand pianos that are moved around a lot are placed on in lieu of individual casters on the legs. They are also called piano trucks.

(That is the on-topic answer to the off-topic question!)

I can't figure out if this is off or on topic, but it's fascinating as heck nonetheless:

NYTimes article about scarf made out of spiderwebs

Wow! It's beautiful. But probably out of my league, price-wise speaking.

Cathy

Spiders are also known as Bots (robots).

Theoretically you can create a robots text file that controls where the spiders "crawl" on your web site.

It works to a certain extent, but doesn't work with all of them (each search engine employs their own sets of bots).

If you click the "Whos Online" link and scroll down you can see which search engines are visiting us at the time. You will often see multiple bots from the same search engine as they send out many of them at a time (it's a big web). Of course sometimes it's just one or two that happen to be crawling all over the forums.

And yes, the search engines do like us because of all the fresh content, and yes, that is a good thing.

The other method for getting a search engines attention is to create a "site map".
Google, Yahoo, and MSN have a particular format they prefer.
Creating a site map helps guide the search engines to the content you want indexed.

Them spiders are doing a great job.
There have been several occasions where my search results have returned the very topic I posted on PW. The first couple times I was at first very surprised, thinking, "WHOA!! Somebody is asking the same question practically word for word!!".

Radiant!

Humble!

Terrific!


Charlotte made me love spiders so much I named my daughter after her.

Hehe I had that reaction too - more along the lines of "WOW! That looks like EXACTLY what I was looking for! ... oh, oops!"



The spider article was fascinating! It doesn't sound terribly economical yet, but the incredibly strength of the spider-silk sounds pretty amazing...

Although... having the silk gently pulled out the spider... errr... doesn't sound so pleasant, for the spider or the person doing the pulling!