dimitris kalamaras

mathematics, social network analysis, free software…

Tag: Web Crawler

SocNetV v1.6 released – a nice web crawler included!

The Social Network Visualizer project has just released its latest version 1.6. This new version brings back the web crawler feature, which has been disabled in the 1.x series so far, but in a much more improved form.

To start the web crawler, go to menu Network > Web Crawler or press Shift+C…

Read More

SocNetV 0.70 – the web crawler release :)

It’s been only one month since the release of 0.60, but Social Network Visualizer version 0.70 is here! This summertime release justifies its existence with two new features, a built-in web crawler and multiple node selection.

The web crawler, based on some older tutorial code I had, is pretty simple software (don’t expect Google power here!), which automatically creates mathematical graphs starting with a given URL, called the seed. As the algorithm crawls that seed webpage, it identifies all the links inside the page and adds them to a list of URLs (called frontier). Then, all the URLs from the frontier are recursively visited. In this process, a network of pages (nodes) and links (edges) is being developed allowing the researcher to study and visualize network properties, such as centralities, etc.

At the moment, the web crawler code is quite immature (i.e. doesn’t recognize framesets) but I hope it will evolve in the future. To test it, go to menu Network > Web Crawler (keyboard junkies, press Shift+C). A dialog will appear – enter the initial web address (seed), the maximum recursion level (how many URLs from the frontier will be visited) and the maximum running time and press OK. Here’s a screenshot of SocNetV crawling http://socnetv.org with betweenness centrality radial layout:

Read More

Creative Commons License
Licensed under a Creative Commons Attribution-ShareAlike 4.0 International License - Powered by Linux