The first open source Web 2.0 website log analyzer, OWS uses PHP and jQuery to provide a powerful and intuitive interface to manipulate website log data stored in a MySQL database. Has a simple plugin system so you can create custom output as well.
This program is called Obsessive Web Statistics for a reason. The idea is to give you a ridiculous amount of control over the web analytics that you retrieve about your website, either for amusement, to improve your site, or for obsessing about how many links you received from Digg yesterday between 12:41am and 3:16pm. Fine-grained multidimensional logfile analysis isn't just possible -- its the whole reason OWS exists!
Things it can do
This program can do many types of analysis on website logs. With how new it is, of course there are some analytics it cannot perform yet. At this point in time, I'm trying to implement unique and useful functionality, with some random features put off until later. Heres an incomplete and ever-growing list of features. Refer to the TODO list (ideas.txt) for possible/planned features.
New! Blog post about practical usage of OWS to analyze traffic spikes
- jQuery-powered Web 2.0 interface
- Daily/Monthly/Yearly/etc Aggregate numbers
- Month-day 'heatmap' (its way cool) for visual analysis of traffic/etc
- Credit: This idea is from Visitors (http://www.hping.org/visitors/)
- Intuititive multi-dimensional logfile analysis
- Identify a lot of bots
- Filter by:
- date and/or time
- HTTP Status codes
- Bot hits/non-bot hits
- Internal/external referrers
- More..
- Support for a lot of different plugins
- Filter plugins
- Option plugins
- Analysis plugins
- Break down large result sets and page through them
- Construct manual SQL-like queries for more detailed analysis
- Show SQL EXPLAIN results to optimize queries (can be turned off)
OWS stores your logfile data in an SQL database, in multiple dimensions for speed and flexibility. This provides you with all the flexibility (and difficulties) that comes with SQL queries and dimensional analysis.
Since much of OWS functionality is implemented in plugins, the plugin interface is designed to be relatively simple to program and integrate common code together where possible to make plugins as simple and powerful as possible. An abstraction layer makes working with the OLAP-style multidimensional tables (relatively) simple for plugin authors. Additionally, you can be sure that the plugin environment actually works well. The plugin is a first class customer on OWS, they aren't just an afterthought that was thrown in. See the plugin development notes for more information.
System Requirements
OWS is designed to be installed on a Linux/Apache/MySQL server, and accessed from a Firefox client. Other combinations of software will probably work. However, it is not a Windows desktop application at this time.
- PHP 5.1 or greater
- Shell access to the machine where you will run OWS
- Access to your raw server logs
- **MySQL, preferably version 5 or greater. It may work on older versions, but I have not done any testing on them
** Ideally, you should be using a table type that uses transactions. I use InnoDB. In fact, the create_table script will warn you if you are not using InnoDB.
Be warned, if you have big log files, then the database can get quite large. For example, check out my table sizes (generated using show_info.php in OWS):
Name Rows Data Idx ========================================================== virtualroadside_com 112111 8.9 MB 32.7 MB virtualroadside_com_agent 1536 196.6 KB 114.7 KB virtualroadside_com_bot 100 16.4 KB 49.2 KB virtualroadside_com_bytes 7260 245.8 KB 163.8 KB virtualroadside_com_config 4 16.4 KB 16.4 KB virtualroadside_com_date 303 16.4 KB 16.4 KB virtualroadside_com_host 5949 278.5 KB 196.6 KB virtualroadside_com_method 5 16.4 KB 16.4 KB virtualroadside_com_protocol 2 16.4 KB 16.4 KB virtualroadside_com_referrer 3121 442.4 KB 491.5 KB virtualroadside_com_request 4131 1.6 MB 917.5 KB virtualroadside_com_status 12 16.4 KB 16.4 KB virtualroadside_com_time 48405 2.6 MB 3.2 MB virtualroadside_com_user 2 16.4 KB 16.4 KB Total Data: 14.4 MB Total Indexes: 37.9 MBThis was with about 50-100 visitors per day for awhile. And this is my site size a few months later with around 10 times more visitors (after a Digg and large amounts of traffic from StumbleUpon):
Name Rows Data Idx =========================================================== virtualroadside_com 1178721 89.8 MB 262.8 MB virtualroadside_com_agent 8436 1.6 MB 540.7 KB virtualroadside_com_bot 159 16.4 KB 49.2 KB virtualroadside_com_bytes 12996 393.2 KB 262.1 KB virtualroadside_com_config 4 16.4 KB 16.4 KB virtualroadside_com_date 335 49.2 KB 16.4 KB virtualroadside_com_host 108034 5.8 MB 7.4 MB virtualroadside_com_method 7 16.4 KB 16.4 KB virtualroadside_com_protocol 3 16.4 KB 16.4 KB virtualroadside_com_referrer 11542 2.6 MB 3.3 MB virtualroadside_com_request 93506 8.9 MB 15.3 MB virtualroadside_com_status 12 16.4 KB 16.4 KB virtualroadside_com_time 86330 3.7 MB 5.3 MB virtualroadside_com_user 2 16.4 KB 16.4 KB Total Data: 112.9 MB Total Indexes: 295.0 MBAs you can see, if you have a very active site, the database can grow extremely large! A 'purging' feature is included, to restrict the amount of data stored in the DB, especially suited for large installations (does not archive old data, yet).
Overall Goals
To provide a powerful yet easy to use statistics analysis framework geared towards analysis of apache log files. Obsessive web stats should allow you to analyze and put every facet of your web traffic under a microscope, and not just deal with aggregate data.
As of this writing, I would say the best open source web statistics analysis software is awstats (http://awstats.sourceforge.net/) -- but we are getting closer each release! It has a ridiculous amount of statistics and such that it can do, and we use it on web sites that we manage. However, it does not provide the level of microscopic detail that we want from web statistics software. Thus, the reason OWS exists.
Similar Programs
There are a lot of programs out there that do website statistics, but there are a few select programs that I would compare OWS to (sorta).
- Awstats (open source, http://awstats.sourceforge.net/)
- Visitors (open source, http://www.hping.org/visitors/)
The following two are both closed-source PHP/MySQL based stats programs:
- Mint (closed source)
- Tracewatch (closed source)
The goal is to not only duplicate and extend the existing functionality in these programs, also add much more advanced functions as well. So yes, there are terms and features directly borrowed from these programs (not source code). Awstats, Visitors, and Mint all support plugin interfaces, though I developed the idea of using plugins before I was aware that they had them available.
OWS Plugin Development
I'm in the process of writing a series of tutorials about how you can write your own plugins for OWS. The first tutorial is posted on my blog. Let me know if you have any questions!
Help OWS
Currently, I am looking for someone who can host a live demo site for OWS. Yes, I do have a demo, but its a very simple one due to my hosting constraints. Contact me if you're interested.