Wednesday, 31 October 2012

Installing Debian unstable safely with APT pinning

Using Aptitude to easily list and install backported or unstable packages

Early into my linux Debian web server administration, out of necessity and after a lot of trail and error and reading I worked out the right preferences that keeps my systems very stable and yet with the ability to install the occasional backported or unstable package. My package management utility of choice is `aptitude` but these settings will also effectively work for `apt-get` and before I begin do +1 this if you find it useful.

The above shows the available version of Django with their respective APT pinning numbers: 990 being the stable version and selected by default when installing, 400 being the backported version, 300 being an unstable version, and 100 pinning for our already installed version.

During normal installation the stable default is always selected, however now I also have an option to easily see other available versions by viewing the package details. Based on the displayed pinning I can tell any backported versions and the unstable version available and review the potential conflicts it could create in a straight forward way.

Setting it all Up

There are four things that need to be edited, the default pinning release needs to be set, the sources need backports and unstable added, lowering the pinning priority of backports/unstable packages, and the aptitude display settings needs to be modified to display pinning.

Apt::default-Release

Create a ' /etc/apt/apt.conf.d/10defaultrelease' and make it's contents as follows:
Apt::default-Release "stable";
.

Update package `sources.list`

Edit your /etc/apt/sources.list to add unstable and backports sources so it looks something like this:

# deb cdrom:[Debian GNU/Linux 6.0.0 _Squeeze_ - Official Multi-architecture amd64/i386 NETINST #1 20110205-14:45]/ squeeze main

deb http://ftp.us.debian.org/debian/ squeeze main
deb-src http://ftp.us.debian.org/debian/ squeeze main

deb http://security.debian.org/ squeeze/updates main
deb-src http://security.debian.org/ squeeze/updates main

# squeeze-update, previously know as 'volatile'
deb http://ftp.us.debian.org/debian/ squeeze-updates main
deb-src http://ftp.us.debian.org/debian/ squeeze-updates main

# squeeze backports
# http://backports.debian.org/Instructions/
deb http://backports.debian.org/debian-backports squeeze-backports main

# unstable
# http://wiki.debian.org/AptPreferences
deb http://ftp.us.debian.org/debian/ unstable main
deb-src http://ftp.us.debian.org/debian/ unstable main

# non free ex. sun java
#deb http://ftp.us.debian.org/debian/ squeeze non-free
#deb-src http://ftp.us.debian.org/debian/ squeeze non-free

Edit `etc/apt/preferences` pinning file

If the file doesn't exist do create it, if it does modify it to add lower pinning the default 500 for unstable and backports packages as follow:

# Package pinning priorities
# See http://wiki.debian.org/AptPreferences and http://manpages.debian.net/cgi-bin/man.cgi?query=apt_preferences
#
# In nut shell highest PIN gets installed
#
# Pining default are as follow which are in addition to our settings:
#   990 - for version that are not installed but DO belong to our `APT::Default-Relase "stable"` setting.
#   500 - for versions that are not installed and do not belong to the target release
#   100 - for packages that already installed, this also means other versions of same package
#     1 - for experimental packages; packages with "NotAutomatic: yes"


Package: *
Pin: release n=squeeze-backports
Pin-Priority: 400

Package: *
Pin: release a=unstable
Pin-Priority: 300

Add pinning information to Aptitude

By default Aptitude does not display any pinning information however this can be changed and once done a pinning number will be displayed when one view the package details, add the following file `/etc/apt/apt.conf.d/`:

Aptitude::UI::Package-Display-Format "%c%a%M %p %Z %v %V %i";

That's it. It is worth pointing that this setup has worked for me over the past 3.5 have years without any issues

It is even able to handle a scenario where a previously installed unstable is upgraded to a stable backported higher version correctly as can be seen with the above example of the Django package.

Wednesday, 19 September 2012

Maximum Logrotate Compression with bzip2

A Tip for squeezing a little more free space from a VPS

KL Insight's Web Design VPS with Linode has been rock a solid solution over the last 4 years. It currently hosts multiple client sites and mail agent and all these sites and emails quickly eat up precious hard drive space; and with only 20GB the wall can be hit early.

The measly 20GB does comes with a kick ass IO rate that beats even some dedicated servers

Stop blabering and get to it

Linux logs everything, these logs grow overtime, systems run 'logrotate' to archive and delete old logs - the settings are very liberal and tweaking them could save gigabytes of space, 1.1GB in my instance. Modify your /etc/logrotate.conf as needed but the important bit is compressing by default and using bzip2 instead of gzip to do so.

# see "man logrotate" for details
# rotate log files weekly
weekly

# keep x weeks worth of backlogs - default 4
rotate 2

# Archive log files by adding a YYYYMMDD instead of simply adding a number.
dateext

# create new (empty) log files after rotating old ones
create

# uncomment this if you want your log files compressed
compress

# use bzip2 which is higher compression the gzip used by default
compresscmd /bin/bzip2
compressoptions -9
compressext .bz2
uncompresscmd /bin/bunzip2

# packages drop log rotation information into this directory
include /etc/logrotate.d

# no packages own wtmp, or btmp -- we'll rotate them here
/var/log/wtmp {
    missingok
    monthly
    create 0664 root utmp
    rotate 1
}

/var/log/btmp {
    missingok
    monthly
    create 0660 root utmp
    rotate 1
}

# system-specific logs may be configured here

There are also specific application/services blog settings in the /etc/logrotate.d directory. Apache by default keeps it's logs for a lengthy 52 weeks! This can be changed by modifying the /etc/logrotate.d/apache2 file as shown below:

/var/log/apache2/*.log {
        weekly
        missingok
        # rotate 52
        compress
        delaycompress
        notifempty
        create 640 root adm
        sharedscripts
        postrotate
                /etc/init.d/apache2 reload > /dev/null
        endscript
}

Be legendary and do social like this post as that helps me immensely.

Tuesday, 11 September 2012

Google Analytics Reverse Lookup

Locating the administrator of a Google Analytics account

For whatever reasons in certain situations a client might have not been granted access to their Google Analytics information by their former developer. This normally is not an issue as developer are mostly a friendly bunch and when asked will grant proper access, but what if the former developer's identity is unknown or lost? - yes rather unlikely scenario but this has happened to us and it can happen to you.

Google Analytics tracking ID reverse lookup by ewhois

On the surface ewhois is your typical look up service with the one and important exception is that it allows tracking down the administrator of analytics tracking code. This can be accomplished due to the fact that site tracking codes contain a common id representing the parent account.

KL Insight Web Design Google Analytics tracking codes are UA-15511708-X where X is the unique id of a tracked site. Based on this fact ewhois is able to return other related sites from it's database

Now with a bit of luck (that the designer is using the same account to track their own site and that ewhois's database contains the information) and some detective work it easy to find the developers business site and their contact information.

Please social this post if it has helped you.

Wednesday, 22 August 2012

Setuptools' include_package_data does not work

MANIFEST.in not honoured during install phase

The python package setup procedure appears to be a mess, you have setuptools, distutils, distutils2 on the way and something new in Python3. All I set out to do was to fix the installation setup.py in django-simplemenu package which was omitting the not python plain locale directory but instead got myself involved a 3 hour roller coaster.

Including sub directories and non python files as part of package

There are two ways: MANIFEST.in file and package_data setup.py option. The latter is Pythonic but doesn’t do recursive includes - so you would have to list every sub directory - and can not include files in the root top most folder; sure once could write a python function to do this but why the complexity? MANIFEST.in with its file/directory pattern matching and with the setup.py's ``include_package_data`` option fits the bill:

``include_package_data``
    If set to ``True``, this tells ``setuptools`` to automatically include any
    data files it finds inside your package directories, that are either under
    CVS or Subversion control, or which are specified by your ``MANIFEST.in``
    file.  For more information, see the section below on `Including Data
    Files`_.

The problem is that does not work, files specified in MAIFEST.in weren't being copied during install. After searching and trial an error the solution was to use distutils instead of the setuptools package. So if you are having issues import distuitls rather then setuptools.

My django-simplemenu setup.py

from distutils.core import setup
from setuptools import find_packages
package = __import__('simplemenu')
setup(name='django-simplemenu',
    version=package.get_version(),
    url='http://github.com/danielsokolowski/django-simplemenu',
    license='BSD',
    description=package.__doc__.strip(),
    author='Alex Vasi , Justin Steward , Daniel Sokolowski ',
    author_email='unemelpmis-ognajd@danols.com',
    include_package_data=True, # this will use MANIFEST.in during install where we specify all of our additional files
    packages=find_packages(),
    # Below is not needed as we are utilizing MANIFEST.in 
    #package_data={'simplemenu': ['locale/en/LC_MESSAGES/*', 
    #                             'locale/ru/LC_MESSAGES/*']
    #              },
    scripts=[],
    requires=[],
    )

My django-simplemenu MANIFEST.in

include LICENSE
include README.rst
recursive-include simplemenu *.py
recursive-include simplemenu/locale *
prune simplemenu/migrations

Tuesday, 21 August 2012

Verify Google+ Local Pages

But old style Google Places pages are still out of luck

Early this month Google introduced functionality to verify the new Google+ Local pages - not the old style Google Places which will be eventually migrated - but the new Google+ pages that must be created under the Local Business or Place category.

The process is simple consists of having a Post Card with a unique pin mailed to your businesse's address. Once that is accomplished your brand page will look something like The Meatball Shop Google+ Page and more importantingly it will be unified accross Google's Search, Google Maps, and Google+.

Business utilizing the former Google Places will be eventually transitioned to Google+ Local. For more information please see the oficial FAQ and discussion threads here and here

Wednesday, 1 August 2012

Unscheduled Secuirty Updates

Web site service might be interrupted for short duration today while we update our web server framework stack to prevent a potential security issues.

We apologize for any inconvenience this might cause and will update on progress as needed.

Monday, 30 July 2012

Open Web Analytics vs Google Analytics

Compairing Open Web Analytics and Google Analytics Part Two

Last time I compared the Dashboard of Open Web Analytics and Google Analytics.

This time I am going to dig into the Dashboard a little more and show some details that I feel make OWA a superior product out of the box to Google Analytics.

Right off the top Open Web Analytics shows traffic graph and visit overview that Google Analytics does. This data is important and certainly will be found in all analytics packages.

The first main difference on the OWA dashboard is the Top Content section (pictured below). This allows you to see what pages are the most 'popular' pages on your site.



As you can see our index page is the page with the most traffic.

You will notice the options at the top for 'Secondary Dimension' and 'Filter'. Filter does exactly what it says it is. Filter allows you to set-up a filter or filters that will allow you to drill down on a specific Secondary Dimension with options like Exactly Matching, Not Matching, Greater than, Less than and Contains.

What are Secondary Dimensions? Well it can be date related such as a specific day, week, month, year, etc Site, Visitor, Visit, System, Netowrk, Campaign, Visit Special, Geo, Content and Custom Vairables. In the image below you can see that I've selected the Operating System of our visitors which is under the 'System Secondary Dimension'.

There is a problem with the options however. It seems that if you select a Dimension that has no data the widget breaks and the page needs to be reloaded.

Like the Top Content getting information on the Top Referrers is just as easy. There is another widget called Top Referrers and it's default view lists the page sending traffic and the number of visits. You can of course filter and apply a Secondary Dimension to the widget to get even more specific data. If you click on one of the referrer pages you will be given even more data on the traffic which includes, a graph of the visits and the date for those visits, The total number of visits, The Avg. Time on site, Unique Visitors, Pages viewed per visit and the bounce rate.




The last widget of note on this page is the Latest Visits widget. This handy digit give you all kinds of great data on your visitors and lets you drill down for even more data.
In the widget you will find the date of the visit and their ISP, a geo location for the visitor, the browser they are using, how any pages they viewed, their visit length, the page they visited and how they found your site. So far we have seen, Direct, organic-search which also includes the Source (Search Engine) and Search Term and finally referral which includes the Source (the referring domain) and a link to the referring page using the page title and listing the actual URL under the link, similar to what the Search Engines Do.



You may notice quickly that some visitors have a 'new' banner over the avatar. I was excited to see this as it means each visitor is remembered and their visits to your site can be tracked. To view the data on the visitor there are two (2) links provided, Visit Detail and Visitor Detail.

The Visit Detail link takes you to a page which displays the same Visit overview that is in the Latest Visits widget for the visitor you are looking at. The page also lists in reverse chronological order the pages on your site the visitor viewed and at what time they viewed the page. The Click Stream as it is called allows you to quickly see the path taken and how long each page was viewed for.



The second option provided Visitor Detail gives you a complete Visitor History which includes the date of the First Visit to your and the total number of visits. This data is followed by the Latest Views Widget display for each visit. This may seem redundant but it allow you to drill down for each individual visit the visitor made allowing you to see their use of the site over time.



OWA captures a lot of data and the Dashboard is intelligently designed to allow the site owner to gather important data quickly and to drill down for even more specific data.

Next time we will look into the Page Detail portion of Open Web Analytics which includes Prior and Next pages viewed, Heatmaps, Domstreams and Dom Clicks.

Friday, 15 June 2012

Open Web Analytics vs Google Analytics

Compairing Open Web Analytics and Google Analytics Part One

Let me start off by saying I am not a huge fan of Google Analytics, never have been. Why? Well it has always seemed clunky to me. I don't if it's the layout, the presentation of data or something else. Even though I am no fan, Google Analytics provides great data expecially considering it is free. It's the defacto standard in Analytics, and it seems that everyone uses it, myself included.

I heard about Open Web Analytics a while back and yesterday I stole some time and got it installed and started tracking our main site to see what kind of data it provides. Well I must say that I am sorry I waited so long to get it installed. I am so impressed with Open Web Analytics, just on the dashboard I find I can get a lot of useful information without having to do a lot of digging.


The Installation:


Getting OWA up and running was a simple matter. Setup a subdomain, create an apahe config file, upload the source, modify the directory and file permissions, setup a database and user (I used phpmyadmin) and go through the OWA setup. Now I ran into a problem with the automated setup so I had to manually create the owa-config file and modify the DB information but it was pretty straight forward. I know it sounds like a lot but it really is quite simple and well documented. I think the whole process took about 10-15 mins to complete and that includes trying to figure out what went wrong when the auto-setup failed.
All said and done, with the documentation if you know how to FTP you can get OWA setup and running. I decided on a sub-domain but there are other options available.

First Impressions:


The intial page wanted me to setup a site to track, which makes sense. So I got it setup and was presented with 2 options, javascript or php tracking. I opted for the traditioanl Javascript code, but having the option to use something else is handy especially if you get a lot of visitors with Javascript disabled.

With the site setup it was time to visit the page to create some data. I should mention that you can exclude IP address from tracking but I wanted to see data now so I skipped that step for now.
Even with only 1 visit what I saw impressed me very much, it looks very similar to GA but there is much more information presented initially.

The Dashboard:

The Dashboard is what makes this software for me. From this page I can get quick information about a lot of useful information. The way in which the data is organized is very clean and allows me to gather an excellent snapshot of visits to the site.

Here is a quick chart compairing the dashboards available in Open Web Analytics and Google Analytics. This data reflects what information is available on the Dashboard without having to click through to any other page.

OWA GA
Traffic Sources Y N
Site Metrics Y Y
Top Content Y N
Visitor Type Y Y
Top Referrers Y N
Latest Visits Y N
Language/Country/City N Y
System: Browser/OS/ISP Y Y
Mobile: OS,Provider,Resolution N Y


Here are some screen shots of the two platforms.

Open Web Analytics Dashboard Google Analytics Dashboard
OWA Dashboard GA Dashboard


Stay tuned as I will be covering more of OWA and GA.

Tuesday, 12 June 2012

VOIP Linphone Incomming Calls Disconecting

Incoming calls terminating after 37 seconds when using Linphone

This fantastic multiple SIP line, multiple platform VOIP soft phone was working great until a week ago where all my incoming calls would terminate after 37 seconds on the dot. Logs were not tell me much. The only indication that was different I could notice was the status icon in the 'My Current Identity' section.

Upon lengthly process of elimination I narrowed it down to an issue with using the UDP as protocol transport for the VOIP calls. Switching it to TCP fixed all my dropped call issues and furthermore the presence status icon is correct; go into your Options > Network Settings > Network Protocols and Ports and switch it to TCP.

I hope this saves someone the same frustration I had as overall this is a great free piece of software.

Thursday, 7 June 2012

Yahoo Site Explorer Reborn!

Best free alternative is the new Bing Link Explorer

It can be argued that the original Yahoo site explorer was the best source for back links, it returned a comprehensive data set compared to Google's 'link:' search modifier.

Current worthy alternative is the Open Site Explorer however the free version limits it's search results and number of daily search

The potentially best new viable alternative is the just added Bing Link Explorer webmaster tool. The most important thing here to be aware of is that it lets you analyze inbound links for sites you are not verified for! Meaning you can analyze the competitor's back links.

However current data set appears to be limited, I believe this will only improve as the tool matures and so I will definitely be keeping my eye on this one.

Tuesday, 29 May 2012

Multiline VOIP Client

Receive call from multiple SIP accounts using one client.

Finally found a decent, free, no hidden costs, and open source VOIP multiline client: Linphone.

It works rather well, doesn't look like dogs breakfast, hasn't crashed yet, and best of all it's also a multi platform client - it will run on your Blackberry, Android, iPhone, Windows and Linux. So in a nut shell if you have multiple sip accounts or run your own PBX system like FreeSWITCH it does the job, and it does it well.

Yes you can thank me, but real thanks should go to Yasuro's Blog, I just used legendary googlefu to find it.

Wednesday, 23 May 2012

Hosting Connectivity Issues

2012-05-23 21:49 - we have received an update that issues are now resolved

We are aware of the network issues with our hosts and are working for a full resolution.

Plain Ctrl+V Paste Text

Windows 7 Plain Paste Keyboard Shortcut

When copying or pasting content Windows attempts to preserve formating between applications. In my line of work this is mostly a hindrance and requires an intermediate Notepad copy and paste step.

However there is hope! and it comes in the form of a small utility named Pure Text. I suggest you place this tiny 28KB program into your Startup folder or on your desktop. Once running you can access its settings through the task tray icon.

In the program's options you can specify a new Windows system wide keyboard shortcut for plain text paste; 'Win' + 'V' seem like a good choice.

Let's hope that Windows 8 will have this built in

Tuesday, 22 May 2012

Fixing Putty Rendering Issues

Configuring Locales and Character Encoding on Linode Debian system

For quite a while I had broken rendering in putty when accessing our Linux servers. Searching the net resulted in solutions most stating to use UTF-8 encoding setting in Putty. This however was only partially successful as line rendering was still broken.

I believe the reason for this is that the Linode Debian installation skips the `locales` configuration bit; to run the setup manually execute the below command:

dpkg-reconfigure locales

Find your locale and choose the 'UTF-8' version of it, for my systems I selected, 'en_CA ISO-8859-1', 'en_CA.UTF-8 UTF-8', 'en_US ISO-8859-1', and 'en_US.UTF-8 UTF-8', with 'en_US.UTF-8' being default. Finally confirm your Putty is using the 'UTF-8' encoding setting in 'Windows > Translation'

Running 'dpkg-reconfigure lecales' again shows that rendering is now correct.

Wednesday, 16 May 2012

DNSChanger Malware

The FBI seems to be concerned about the DNSChanger Malware that is infecting machines around the world. You might want to visit the site http://www.dcwg.org/ prior to July 9th and test your system or you might lose your internet connectivity. This site has a lot of information about this malware and also provides links to test your system and if infected directs you on how to remove the malware.

The FBI also has a site http://www.fbi.gov/news/stories/2011/november/malware_110911 that discusses DNSChanger.  

I would say this is a legitimate concern.   I have only tested my machine so far and it tested clean.

While on the topic of virus / malware attacks, have you given thought to a disaster recovery plan?    First step is prevention, with trusted antivirus software.   Even more important is a tested backup / recovery plan.   And the final step is  a plan that you would follow in the event of physical disaster, whether it be an office fire, theft of systems, or hardware failure.

Don't learn the hard way!   Give disaster planning some thought, and if you need any help with this please contact us to assist you.

Update (May 23, 2012)

You will see that in this article (http://news.cnet.com/8301-1009_3-57439407-83/google-will-alert-users-to-dnschanger-malware-infection/) Google will be warning people that their computer is infected.   If you get a notice (see below) on your browser when searching then you will need to follow the directions for removal.

The warning that will appear at the top of search results for people whose computers are infected.





Monday, 30 April 2012

Apache Prefork vs Worker

Lower memory usage by 22% and higher performance with apache2-mpm-worker

Apache Multi-Processing Modules (MPM) prefork is the original like apache 1.3 implementation which is used where multi-threading could cause issues i.e. PHP hosting. The worker MPM is a multi-threaded and claims to provide higher performance and lower memory footprint than prefork.
When the time came to update one of our hosting servers, we decided to included the move to the worker mpm implementation, I tested the performance mpm implementation and found that it achieved a savings of 17.4MiB - 20.8 MiB and 8 less child processes:

Apache MPM Prefork

After clean apache reboot:
    $ ./ps_mem.py
    Private  +   Shared  =  RAM used       Program
    64.7 MiB +  11.8 MiB =  76.5 MiB       apache2 (23)
After each of our sites was accessed once:
    $  ./ps_mem.py
    Private  +   Shared  =  RAM used       Program
    245.8 MiB +  12.8 MiB = 258.6 MiB       apache2 (31)

Apache MPM Worker

Results with apache2-mpm-worker module after a clean apache reboot:
    $  ./ps_mem.py
    Private  +   Shared  =  RAM used       Program
    53.1 MiB +   6.0 MiB =  59.1 MiB       apache2 (20)
After each site was hit once with an web browser:
    $  ./ps_mem.py
    Private  +   Shared  =  RAM used       Program
    225.7 MiB +  12.1 MiB = 237.8 MiB       apache2 (21)
From this semi scientific test it is clearly worthwhile using apache2-mpm-worker on a server that is purely serving and hosting django projects.

Friday, 27 April 2012

Free 25 GB Cloud Storage

SkyDrive offering free upgrade to existing users

Recently Microsoft has updated their SkyDrive cloud storage solution, however with this update they also reduced storage to 7 GB. Existing SkyDrive can upgrade 'back' to their 25 GB free storage by visiting their account manage page. Please note you must have used the service before this upgrade to qualify, see this page for more details.

It is worth pointing out that even the reduced 7 GB beats Google Drive and Apple iCloud by 2 GB and DropBox by 5 GB. More information can be found on the official SkyDrive compare page.

Further more one can upgrade up to 100 GB of storage $50.00/year making it a feasible cheaper alternative to cloud backup/archive offered by S3 Amazon Storage. This task can be accomplished by setting up FUSE (Filesystem in Userspace) which is a post for another day.

Wednesday, 25 April 2012

Welcome

I would like to welcome everyone who finds there way to this blog.  Our team at KL Insight will be posting information we feel is relevant to the ever-changing world of Web Design and Search Engine Optimization.

We pride ourselves on staying current and would like to pass on our finds to our followers.

Regards,

Mark Pizzinato
President, KL Insight