Friday, 19 April 2013

mppe_compress[0]: osize too small! (have: 1404 need: 1408)

Windows VPN and PPTP Connectivity Timeouts

In my pursuit for a personal cloud and convenient secure remote file system so as to be able to develop directly on the server I have researched, tested and decided against SFTP, WebDAV, and SMB/NFS over internet. The only approach still standing is the built in Windows VPN and PPTP + SAMBA/CIFS:


However there was one bug that prevented some sites/services from working and timing out; the /var/log/syslog showed the following intermittently:

Apr 19 10:15:15 sh1 pptpd[11330]: GRE: accepting packet #107
Apr 19 10:15:15 sh1 pptpd[11330]: GRE: accepting packet #108
Apr 19 10:15:15 sh1 pptpd[11330]: GRE: accepting packet #109
Apr 19 10:15:15 sh1 kernel: mppe_compress[0]: osize too small! (have: 1404 need: 1408)
Apr 19 10:15:15 sh1 kernel: ppp0: ppp: compressor dropped pkt
Apr 19 10:15:15 sh1 kernel: mppe_compress[0]: osize too small! (have: 1404 need: 1408)

This due to the way MPPE Microsoft point-to-point Encryption encodes data which results in the packet size being bigger then what was agreed in the VPN handshake - is my guess. There is a reported bug from 2005 which sadly hast not yet been addressed.

Fixing the issue by increasing the MTU

You can't fix this issue by modifying the MTU/MRU settings in '/etc/ppp/options' directly, you have to adjust the MTU after the PPP connection is up and this can be accomplished by adding a custom 'ip-up' script. Below is my work around script, place it into file '/etc/ppp/ip-up.d/mppefixmtu' and ensure that it is executable ('chmod +x mppefixmtu'):

#!/bin/sh
CURRENT_MTU="`ifconfig $1 | grep -Po '(?<=MTU:)([0-9]+)'`"
FIXED_MTU="`expr $CURRENT_MTU + 4`"
ifconfig $1 mtu $FIXED_MTU
echo "Increased MTU for $1 to $FIXED_MTU (from $CURRENT_MTU) to fix MPPE Microsoft Point-to-Point bug #330973"

Troubleshooting

You can review script errors by examining the file '/var/log/ppp-ipupdown.log', however if the file doesn't exist then you must create it to enable ip-up/ip-down script logging - don't forget to restart pppd.

As always if you found this useful feel free to follow me here or via twitter @danielsokolowski.

Tuesday, 26 March 2013

Automatically Refresh Browser Window

Automatically refresh the browser window when saving a file in Eclipse IDE

The edit/preview workflow pattern is make some change, save the files (CTR+S), swtich to browser (ALT+Tab), refresh the windows (F5), switch back to Eclipse IDE (ALT+Tab) and rinse and repeat. Even though I can do this quickly it is repetitive and cumbersome and the seconds it takes turn into minutes and hours wasted over many projects.

Googling you will find posts that show how to automate the Alt+Tab F5 Alt+Tab sequence using VBScript and setting it as an automatic builder in Eclipse.


Now this is a good enough solution and it does work very well except that it does not work with all browsers. Google Chrome being one of them where the SendKeys seem to be ignored. However using an alternative utility NirCmd I was able to make it work as follows:
  • Download NirCmd and extract 'nircmd.exe' into your projects.
  • Create the following 'nircmd-chrome-focus+f5.bat' batch file:

    ECHO Off
    REM This little batch files calls the awesome nircmd utility to focus Chrome window send an F5 and swtich
    REM back to Eclipse. This is no longer possible with VBScript in WIN7 as Chrome can only be focused but won't 
    REM accept key sends unless a click is made. Seriously donate to the NirSoft for making this tool. 
    
    %~dp0nircmd.exe win activate ititle "- Google Chrome"
    %~dp0nircmd.exe win max ititle "- Google Chrome"
    %~dp0nircmd.exe sendkey f5 press
    %~dp0nircmd.exe win activate ititle "- Eclipse"
    %~dp0nircmd.exe win max ititle "- Eclipse" 
    
    
  • Add the batch file as a project builder 'Project > Properties > Builder > New' and ensure 'Project > Build Automatically' is selected. See above screen shot for assistance.
Feel free to comment or share this if it has hepled you, also consider donating to NirSoft and check out some of their other utilities.

Thursday, 7 February 2013

Windows WebDAV Double Authentication 401 Requets

Speed up trick for Windows 7 WebDAV Client

One way to conveniently develop remotely with Eclipse + PyDev is to run WebDAV on the remote server and map it as a networked drive in Windows (PyDev over Eclipse Target Managmenet does not play nicely). WebDAV works rather well except it is not the fastest especially on initial project creation /project refreshing and if your share is protected with a username/password; below is how I improved the performance.

Apparently by protocol design and due to security Windows 7 client does not cache it's authentication and so on each request first an anonymous attempt is made followed by an authenticated one:

...
72.38.184.18 - - [22/Jan/2013:23:05:04 +0000] "PROPFIND /Files HTTP/1.0" 401 751
72.38.184.18 - username [22/Mar/2011:23:05:04 +0000] "PROPFIND /Files HTTP/1.0" 301 495
72.38.184.18 - - [22/Jan/2013:23:05:04 +0000] "PROPFIND /Files/ HTTP/1.0" 401 751
72.38.184.18 - username [22/Mar/2011:23:05:04 +0000] "PROPFIND /Files/ HTTP/1.0" 207 1175
72.38.184.18 - - [22/Jan/2013:23:05:07 +0000] "PROPFIND /Files HTTP/1.0" 401 751
...

After a bit of research I was able to implement a work around, allow anonymous access to the PROPFIND requests which are mainly used for directory listings and authenticated access for everything else. Refreshing a python/djagno project will take half as long for example 21876 files took 11 minutes with no authentication vs 21 with authentication

## Development HTTP Site
<VirtualHost *:80>
  ServerAdmin FOO@FOO.com
  ServerName development.FOO.com
  ServerAlias www.development.FOO.com

  # Log file location and settings; logs within project is ok as long as 'links' are made to system 'var/log/apache'
  ErrorLog /var/log/apache2/development.FOO.com-error.log
  CustomLog /var/log/apache2/development.FOO.com-vhost_combined-access.log vhost_combined

  # Canonical to always strip www - see: http://stackoverflow.com/questions/88011/make-apache-automatically-strip-off-the-www
  RewriteCond %{HTTP_HOST} ^www\.(.+)$
  RewriteRule ^(.*)$ ${SERVER_PROTOCOL}://%1/$1 [R=301,L,NC]

  # Authenticated access for the development site version - because without this Google will find you!
  # Just in case we also prevent serving of the password logins file if it is stored in a serving folder.
  Redirect /apache-logins.htdigest http://development.FOO.com
  <Location />
    AuthType Digest
    AuthName "development.FOO.com"
    # AuthDigestDomain which urls (and any under it) this applies - should match location
    AuthDigestDomain /
    AuthDigestProvider file
    AuthUserFile /srv/www/django/development.FOO.com/apache-logins.htdigest
    # uncomment the LimitExcept to receive a small boost for non caching Windows WebDAV client by allowing
    # anonymous directory listing; see http://serverfault.com/questions/250578/webdav-and-windows-7-client
    <LimitExcept PROPFIND>
     Require valid-user
    </LimitExcept>
  </Location>

  WSGIProcessGroup development.FOO.com
  # You can further limit processes, threads and set a inactivity-timer so deamon get unloaded
  WSGIDaemonProcess development.FOO.com display-name=%{GROUP}
  WSGIScriptAlias / /srv/www/django/development.FOO.com/apache-django-development.wsgi

  # Serve static / media files through apache instance and alias/map them to specific urls. to maximize security
  # `Options -Indexes` is enabled to prevent directory listing
  Options -Indexes
  Alias /robots.txt /srv/www/django/development.FOO.com/src/django-project/static/robots.txt
 #Alias /sitemap.xml /srv/www/django/development.FOO.com/src/django-project/static/sitemap.xml
  Alias /favicon.ico /srv/www/django/development.FOO.com/src/django-project/static/favicon.ico
  Alias /media /srv/www/django/development.FOO.com/src/django-project/static/
  Alias /static /srv/www/django/development.FOO.com/src/django-project/static/

</VirtualHost>

Hope this helps you out, feel free to follow me on twitter or google plus: @danielsokolowski and +1 this.

Wednesday, 31 October 2012

Installing Debian unstable safely with APT pinning

Using Aptitude to easily list and install backported or unstable packages

Early into my linux Debian web server administration, out of necessity and after a lot of trail and error and reading I worked out the right preferences that keeps my systems very stable and yet with the ability to install the occasional backported or unstable package. My package management utility of choice is `aptitude` but these settings will also effectively work for `apt-get` and before I begin do +1 this if you find it useful.

The above shows the available version of Django with their respective APT pinning numbers: 990 being the stable version and selected by default when installing, 400 being the backported version, 300 being an unstable version, and 100 pinning for our already installed version.

During normal installation the stable default is always selected, however now I also have an option to easily see other available versions by viewing the package details. Based on the displayed pinning I can tell any backported versions and the unstable version available and review the potential conflicts it could create in a straight forward way.

Setting it all Up

There are four things that need to be edited, the default pinning release needs to be set, the sources need backports and unstable added, lowering the pinning priority of backports/unstable packages, and the aptitude display settings needs to be modified to display pinning.

Apt::default-Release

Create a ' /etc/apt/apt.conf.d/10defaultrelease' and make it's contents as follows:
Apt::default-Release "stable";
.

Update package `sources.list`

Edit your /etc/apt/sources.list to add unstable and backports sources so it looks something like this:

# deb cdrom:[Debian GNU/Linux 6.0.0 _Squeeze_ - Official Multi-architecture amd64/i386 NETINST #1 20110205-14:45]/ squeeze main

deb http://ftp.us.debian.org/debian/ squeeze main
deb-src http://ftp.us.debian.org/debian/ squeeze main

deb http://security.debian.org/ squeeze/updates main
deb-src http://security.debian.org/ squeeze/updates main

# squeeze-update, previously know as 'volatile'
deb http://ftp.us.debian.org/debian/ squeeze-updates main
deb-src http://ftp.us.debian.org/debian/ squeeze-updates main

# squeeze backports
# http://backports.debian.org/Instructions/
deb http://backports.debian.org/debian-backports squeeze-backports main

# unstable
# http://wiki.debian.org/AptPreferences
deb http://ftp.us.debian.org/debian/ unstable main
deb-src http://ftp.us.debian.org/debian/ unstable main

# non free ex. sun java
#deb http://ftp.us.debian.org/debian/ squeeze non-free
#deb-src http://ftp.us.debian.org/debian/ squeeze non-free

Edit `etc/apt/preferences` pinning file

If the file doesn't exist do create it, if it does modify it to add lower pinning the default 500 for unstable and backports packages as follow:

# Package pinning priorities
# See http://wiki.debian.org/AptPreferences and http://manpages.debian.net/cgi-bin/man.cgi?query=apt_preferences
#
# In nut shell highest PIN gets installed
#
# Pining default are as follow which are in addition to our settings:
#   990 - for version that are not installed but DO belong to our `APT::Default-Relase "stable"` setting.
#   500 - for versions that are not installed and do not belong to the target release
#   100 - for packages that already installed, this also means other versions of same package
#     1 - for experimental packages; packages with "NotAutomatic: yes"


Package: *
Pin: release n=squeeze-backports
Pin-Priority: 400

Package: *
Pin: release a=unstable
Pin-Priority: 300

Add pinning information to Aptitude

By default Aptitude does not display any pinning information however this can be changed and once done a pinning number will be displayed when one view the package details, add the following file `/etc/apt/apt.conf.d/`:

Aptitude::UI::Package-Display-Format "%c%a%M %p %Z %v %V %i";

That's it. It is worth pointing that this setup has worked for me over the past 3.5 have years without any issues

It is even able to handle a scenario where a previously installed unstable is upgraded to a stable backported higher version correctly as can be seen with the above example of the Django package.

Wednesday, 19 September 2012

Maximum Logrotate Compression with bzip2

A Tip for squeezing a little more free space from a VPS

KL Insight's Web Design VPS with Linode has been rock a solid solution over the last 4 years. It currently hosts multiple client sites and mail agent and all these sites and emails quickly eat up precious hard drive space; and with only 20GB the wall can be hit early.

The measly 20GB does comes with a kick ass IO rate that beats even some dedicated servers

Stop blabering and get to it

Linux logs everything, these logs grow overtime, systems run 'logrotate' to archive and delete old logs - the settings are very liberal and tweaking them could save gigabytes of space, 1.1GB in my instance. Modify your /etc/logrotate.conf as needed but the important bit is compressing by default and using bzip2 instead of gzip to do so.

# see "man logrotate" for details
# rotate log files weekly
weekly

# keep x weeks worth of backlogs - default 4
rotate 2

# Archive log files by adding a YYYYMMDD instead of simply adding a number.
dateext

# create new (empty) log files after rotating old ones
create

# uncomment this if you want your log files compressed
compress

# use bzip2 which is higher compression the gzip used by default
compresscmd /bin/bzip2
compressoptions -9
compressext .bz2
uncompresscmd /bin/bunzip2

# packages drop log rotation information into this directory
include /etc/logrotate.d

# no packages own wtmp, or btmp -- we'll rotate them here
/var/log/wtmp {
    missingok
    monthly
    create 0664 root utmp
    rotate 1
}

/var/log/btmp {
    missingok
    monthly
    create 0660 root utmp
    rotate 1
}

# system-specific logs may be configured here

There are also specific application/services blog settings in the /etc/logrotate.d directory. Apache by default keeps it's logs for a lengthy 52 weeks! This can be changed by modifying the /etc/logrotate.d/apache2 file as shown below:

/var/log/apache2/*.log {
        weekly
        missingok
        # rotate 52
        compress
        delaycompress
        notifempty
        create 640 root adm
        sharedscripts
        postrotate
                /etc/init.d/apache2 reload > /dev/null
        endscript
}

Be legendary and do social like this post as that helps me immensely.

Tuesday, 11 September 2012

Google Analytics Reverse Lookup

Locating the administrator of a Google Analytics account

For whatever reasons in certain situations a client might have not been granted access to their Google Analytics information by their former developer. This normally is not an issue as developer are mostly a friendly bunch and when asked will grant proper access, but what if the former developer's identity is unknown or lost? - yes rather unlikely scenario but this has happened to us and it can happen to you.

Google Analytics tracking ID reverse lookup by ewhois

On the surface ewhois is your typical look up service with the one and important exception is that it allows tracking down the administrator of analytics tracking code. This can be accomplished due to the fact that site tracking codes contain a common id representing the parent account.

KL Insight Web Design Google Analytics tracking codes are UA-15511708-X where X is the unique id of a tracked site. Based on this fact ewhois is able to return other related sites from it's database

Now with a bit of luck (that the designer is using the same account to track their own site and that ewhois's database contains the information) and some detective work it easy to find the developers business site and their contact information.

Please social this post if it has helped you.

Wednesday, 22 August 2012

Setuptools' include_package_data does not work

MANIFEST.in not honoured during install phase

The python package setup procedure appears to be a mess, you have setuptools, distutils, distutils2 on the way and something new in Python3. All I set out to do was to fix the installation setup.py in django-simplemenu package which was omitting the not python plain locale directory but instead got myself involved a 3 hour roller coaster.

Including sub directories and non python files as part of package

There are two ways: MANIFEST.in file and package_data setup.py option. The latter is Pythonic but doesn’t do recursive includes - so you would have to list every sub directory - and can not include files in the root top most folder; sure once could write a python function to do this but why the complexity? MANIFEST.in with its file/directory pattern matching and with the setup.py's ``include_package_data`` option fits the bill:

``include_package_data``
    If set to ``True``, this tells ``setuptools`` to automatically include any
    data files it finds inside your package directories, that are either under
    CVS or Subversion control, or which are specified by your ``MANIFEST.in``
    file.  For more information, see the section below on `Including Data
    Files`_.

The problem is that does not work, files specified in MAIFEST.in weren't being copied during install. After searching and trial an error the solution was to use distutils instead of the setuptools package. So if you are having issues import distuitls rather then setuptools.

My django-simplemenu setup.py

from distutils.core import setup
from setuptools import find_packages
package = __import__('simplemenu')
setup(name='django-simplemenu',
    version=package.get_version(),
    url='http://github.com/danielsokolowski/django-simplemenu',
    license='BSD',
    description=package.__doc__.strip(),
    author='Alex Vasi , Justin Steward , Daniel Sokolowski ',
    author_email='unemelpmis-ognajd@danols.com',
    include_package_data=True, # this will use MANIFEST.in during install where we specify all of our additional files
    packages=find_packages(),
    # Below is not needed as we are utilizing MANIFEST.in 
    #package_data={'simplemenu': ['locale/en/LC_MESSAGES/*', 
    #                             'locale/ru/LC_MESSAGES/*']
    #              },
    scripts=[],
    requires=[],
    )

My django-simplemenu MANIFEST.in

include LICENSE
include README.rst
recursive-include simplemenu *.py
recursive-include simplemenu/locale *
prune simplemenu/migrations