## Tuesday, November 4, 2014

### Using LPSOLVE for linear programming problems.

Although Gnumeric and LibreOffice CALC spreadsheet softwares can solve linear programming problems, these do not include postoptimality or sensitivity reports. But not to worry.

The free standing LP program LPSOLVE can be used to solve linear programming problems and is capable of producting sensitity reports.
Let us illustrate how to use this software on a simple program.

Create a file, simple.py with the following contents:

```max: 5000 X+ 3000 Y;

C1: X + Y   <=  8;
C2: 2 X + Y <= 10;
```
Note that LPSOLVE will do the automatic conversion of the above problem to a standard from internally, the user need not explicity rewrite the model to include slack, surplus or artificial variables. From the command line, issue lp_solve -S7 simple.py . This will tell the program to solve the lp problem specification and solve for the optimum values of the variables and to produce a complete report :
```Model name:
X        Y
Maximize      5000     3000
C1               1        1 <=        8
C2               2        1 <=       10
Type          Real     Real
upbo           Inf      Inf
lowbo            0        0

Scale factors:
R0                   scaled at 0.000237841
C1                   scaled at 0.840896
C2                   scaled at 0.594604
X                    scaled at 0.840896
Y                    scaled at 1.18921

Value of objective function: 28000

Actual values of the variables:
X                               2
Y                               6

Actual values of the constraints:
C1                              8
C2                             10

Objective function limits:
From            Till       FromValue
X                                3000            6000          -1e+30
Y                                2500            5000          -1e+30

Dual values with from - till limits:
Dual value            From            Till
C1                               1000               5              10
C2                               2000               8              16
X                                   0          -1e+30           1e+30
Y                                   0          -1e+30           1e+30

Tableau at iter 2:
-3             -4
2      2.0000000     -1.4142136      5.0453785
1     -1.4142136      2.0000000      2.3784142
-0.2828427     -0.8000000      6.6595598

```
The output contains the model formulation, scaling factors used, value of objective function(28000), value of the variables,X=2 ad Y=6 constraint values (value of slacks), and the final final tableu at termination If there were errors in the specification file, the software may not output the reusults. Thee flag -S7 gives the most detailed output. Here are other flag setttings:
```-S0: Print nothing
-S1: Only objective value
-S2: Obj value+variables (default)
-S3: Obj value+variables+constraints
-S4: Obj value+variables+constraints+duals
-S5: Obj value+variables+constraints+duals+lp model
-S6: Obj value+variables+constraints+duals+lp model+scales
-S7: Obj value+variables+constraints+duals+lp model+scales+lp tableau

```
It is beyond the scope of this article to explain or interpret the output values retuned by LPSOLVE. That might be explained in another artcle in our other blogs.

## Friday, May 11, 2012

### The Ceres nonlinear least squares solver from Google. Part 1. Installation.

#### Introduction

Ceres is a new C++ library and optimization software for nonlinear least squares problems. It is aimed for fitting models in statistics and finds use in computer vision at Google. It was released as an open source software with BSD license by Google and was developed by Sameer Aggarwal and Keir Mierle. The home page is at
http://code.google.com/p/ceres-solver/. The following are features of Ceres:

• A friendly API: build your cost function one residual at a time
• Automatic differentiation
• Robust loss functions
• Local parameterizations
• Threaded Jacobian evaluators and linear solvers
• Dense QR factorization (using Eigen) for small problems
• Sparse Cholesky factorization (using SuiteSparse) for large sparse problems
• Specialized solvers for bundle adjustment problems in computer vision.
• Scales from servers to cell phones.
• Supports Linux and MacOS with future support for Windows and iOS and Android.
We are surprised that the instructions for installing Ceres software is in a pdf file in documentation of the installer. So we read the documentation and publish the instructions which we followed and seen to work.

#### Installing Ceres using a terminal

1. Create a subdirectory ceres-1.0 and cd to that created directory. 2. Download using wget.
```\$mkdir ceres-1.0
\$ cd ceres-1.0
Resolving ceres-solver.googlecode.com... 209.85.175.82 Connecting to ceres-solver.googlecode.com|209.85.175.82|:80... connected. HTTP request sent, awaiting response... 200 OK
Length: 3461260 (3.3M) [application/x-gzip] Saving to: `ceres-solver-1.0.0.tar.gz'  100%[====================================================================================>]
3,461,260   65.1K/s   in 54s       2012-05-11 15:10:47 (62.2 KB/s) - `ceres-solver-1.0.0.tar.gz' saved [3461260/3461260]
```
3. Untar the gzipped file by typing tar xzvvf ceres-solver-1.0.0.tar.gz. Change the current directory to ceres-solver-1.0.0.

#### Software Dependencies

4. Install various dependency files. You need to install the following softwares in order for Ceres to do its thing: cmake, gflags, glog, eigen , suitesparse and protobuf:
```sudo apt-get install cmake
tar xzvvf gflags-2.0.tar.gz   cd gflags-2.0
./configure --prefix=/usr/local
make
cd ..
tar xzvvf glog-0.3.2.tar.gz
cd glog-0.3.2/
./configure --with-gflags=/usr/local/
make
sudo make install
sudo apt-get install libeigen3-dev
sudo apt-get install libsuitesparse-dev
sudo apt-get install libprotobuf-dev
```

#### 5. Now it is time to install the Ceres software.

```mkdir ceres-bin
cd ceres-bin
cmake ../ceres-solver-1.0.0
cmake ../ceres-solver-1.0.0 -DEIGEN_INCLUDE=/usr/include/eigen3
make -j3
```

#### Example

6. Running the example:
```examples/simple_bundle_adjuster ../ceres-solver-1.0.0/data/problem-16-22106-pre.txt
0: f: 1.598216e+06 d: 0.00e+00 g: 5.67e+18 h: 0.00e+00 rho: 0.00e+00 mu: 1.00e-04 li:  0
1: f: 1.116401e+05 d: 1.49e+06 g: 1.42e+18 h: 5.48e+02 rho: 9.50e-01 mu: 3.33e-05 li:  1
2: f: 4.923547e+04 d: 6.24e+04 g: 8.57e+17 h: 3.21e+02 rho: 6.79e-01 mu: 3.18e-05 li:  1
3: f: 1.884538e+04 d: 3.04e+04 g: 1.45e+17 h: 1.25e+02 rho: 9.81e-01 mu: 1.06e-05 li:  1
4: f: 1.807384e+04 d: 7.72e+02 g: 3.88e+16 h: 6.23e+01 rho: 9.57e-01 mu: 3.53e-06 li:  1
5: f: 1.803397e+04 d: 3.99e+01 g: 1.35e+15 h: 1.16e+01 rho: 9.99e-01 mu: 1.18e-06 li:  1
6: f: 1.803390e+04 d: 6.16e-02 g: 6.69e+12 h: 7.31e-01 rho: 1.00e+00 mu: 3.93e-07 li:  1

Ceres Solver Report
-------------------
Original                  Reduced
Parameter blocks                        22122                    22122
Parameters                              66462                    66462
Residual blocks                         83718                    83718
Residual                               167436                   167436

Given                     Used
Linear solver                     DENSE_SCHUR              DENSE_SCHUR
Preconditioner                            N/A                      N/A
Ordering                                SCHUR                    SCHUR
num_eliminate_blocks                      N/A                    22106

Cost:
Initial                          1.598216e+06
Final                            1.803390e+04
Change                           1.580182e+06

Number of iterations:
Successful                                  6
Unsuccessful                                0
Total                                       6

Time (in seconds):
Preprocessor                     1.000000e+00
Minimizer                        5.000000e+00
Total                            6.000000e+00
Termination:               FUNCTION_TOLERANCE

```
In the next part, we explain the example and try to understand the API for using the Ceres Software.

## Thursday, April 5, 2012

### Create bootable "live" LInux or BSD usb drives with unetbootin

Unetbootin is a powerful system utility to create bootable usb drives with your favorite distribution(although may not be the latest version). You need at least a usb flash drive with 2Gbytes capacity, but it is better to use 4 Gbytes, the price difference is small enough.

You also need gparted utility to format and partition your drive. If these are not available in your LInux OS, install them using your software package installer system or if you have moderate skills, download, compile and install from sources. You can install apt-get and synaptic, we wonder why Ubuntu removed this from their newest distribution.

Insert your usb drive, saving important files to your hard disk. Then use gparted to format using "ext2" if you want to try Puppy LInux. Otherwise you can also format using MS FAT32 so you can even save files readable in Windows!

Issue df in the command line after using gparted. Your usb should be detected, usually the drive is accessible using the name /media. For example, my pqi branded usb is readble/writable as /media/d32b6e24-c9dd-4073-8f24-63981d635de1.

Next issue sudo unetbootin. You will be presented with subwindow giving you choices of Operating Systems, including BSDs!(freebsd and netbsd.). Yes you can use the internet to download the operating system ISO images, but it would be masochism as realtime downloading is a pain! Instead copy the iso link using the mouse and use wget instead. In fact it tooke me more than two hours to download a live Linux Mint iso image!

Here is my wget for free bsd. Mercifully it is a small size,
wget -ct 0 http://nchc.dl.sourceforge.net/project/lubi/Zipped%20Disk%20Images/1/freebsd-8.0.img.gz

For puppy, use

wget -ct 0 http://distro.ibiblio.org/pub/linux/distributions/puppylinux/puppy-5.3.1/slacko-5.3.1-SCSI-MAIN.iso

Here is a screenshot of unetbootin:

Note that your usb must be mounted when you use unetbootin. Click on the small button with .... Then hightlight the location of your downloaded iso. Then click on ok button. Wait a while and unetbootin will tell you when it is safe to reboot!

I have done this puppy and linuxmint and freebsd. I find the bsd distribution difficult to install, it needs more information! Here are some screenshots of opening screen for puppy:

And here is for LinuxMint.

Pupppy linux is fast to boot, but it is so slow shutting down! It incorporates the seamonkey browser.
As for LInuxmint,it uses a very old version of Firefox, 3.5.0, the newest version is already at 12.0! Oh well, we cannot have everything.

Itis surprising that Knoppix, the pioneering live-CD Linux distribution is not in the list of unetbootin!

## Wednesday, April 4, 2012

### Will Spyder be our new Python IDE?

Your author has tried Emacs, Vi, eric4, drpython, spe, gedit, jedit, anjuta,idle, geany
kate, kedit, when writing Python programs, or modules

Among the editors/IDE mentioned eric4 comes close to be the most ideal for me, except that
is somewhat slow. Spe on the otherhand, has that unusual cursor-editor missynchronization.What
character you type sometimes does not appear immediately! It is disconcerting to say the least.

Here is a short list of features I demand to have in an editor:

1. Quick zooming, unzooming. I wonder why editors do not implement or have removed
the standard browser Ctrl-plus or Ctrl-minus for easy viewing of source codes?

2. Interactive debugging support.

3. Indent/unindent highlighted code blocks.

4. Folding and unfolding code blocks. Not an absolute requirement.
Spyder aims to be the standard scientific python editor/ide. It is available in Ubuntu apt-get but it refused to run due to some versioning errors. One day last week, I got an email announcement from the scipy mailing list that there is a new version of spyder. So we tried installing it from the sources.Here are the steps:
```wget -ct 0 http://spyderlib.googlecode.com/files/spyder-2.1.9.zip
unzip spyder-2.1.9.zip
cd spyder-2.1.9/
sudo python2.7 setup.py install
spyder
```
That works! and that is the latest version of spyder. Spyder on default mode opens the following windows in one screen: console,editor and a tabbed inspector explorer/window for objects, variables, and files. Typing scientific on the console window gave the following information:
```>>> scientific
This is a standard Python interpreter with preloaded tools for scientific
computing and visualization:

>>> import numpy as np  # NumPy (multidimensional arrays, linear algebra, ...)
>>> import scipy as sp  # SciPy (signal and image processing library)

>>> import matplotlib as mpl         # Matplotlib (2D/3D plotting library)
>>> import matplotlib.pyplot as plt  # Matplotlib's pyplot: MATLAB-like syntax
>>> from pylab import *              # Matplotlib's pylab interface
>>> ion()                            # Turned on Matplotlib's interactive mode

Within Spyder, this intepreter also provides:
* special commands (e.g. %ls, %pwd, %clear)
* system commands, i.e. all commands starting with '!' are subprocessed
(e.g. !dir on Windows or !ls on Linux, and so on)

>>>
```

We will spend more time with our newest editor/ide, but spyder looks very, very promising.

## Sunday, March 25, 2012

### AlgoPy-automatic differentiation for Python programs

Note: There was failure in running the example in my system consisting of Python 2.7 and the 0.3.1 version of AlgoPy.

#### Introduction

I had a problem determining the gradient of the ordinal logistic model and I could not find online references on the formula for the gradient. You see it takes around 6 seconds to compute the parameters of model using the BFGS (Broyden, Fletcher, Goldfarb, Shanno) optimizer.

I have heard of ADOL-C before for automatic differentiation, in fact I was browsing the second hand Book Sale and I found a reference book with Java-Forte programs on a CD. That turned me off, since Forte is relative old IDE and I decided NOT to buy it. So lets try an automatic differentiation package for Python.

The documentation at the homepage states the purpose of AlgoPy nicely:

The purpose of AlgoPy is the evaluation of higher-order derivatives in the forward and reverse mode of Algorithmic Differentiation (AD) of functions that are implemented as Python programs. Particular focus are functions that contain numerical linear algebra functions as they often appear in statistically motivated functions. The intended use of AlgoPy is for easy prototyping at reasonable execution speeds. More precisely, for a typical program a directional derivative takes order 10 times as much time as time as the function evaluation. This is approximately also true for the gradient.

#### Capabilities of AlgoPy

AlgoPy promises to compute the following if possible the following:

• evaluation of derivatives useful for nonlinear continuous optimization
• Jacobian
• Hessian
• Jacobian vector product
• vector Jacobian product
• Hessian vector product
• vector Hessian vector product
• higher-order tensors
• Taylor series evaluation

#### Installation

I am using the latest Ubuntu OS ( march 24, 2012
Oneiric Ocelot) on my laptop which serves as my main computer. Try to see the version of your installed default Python by issuing python --version. I see that I have 2.7.2+ Now try this:
`sudo easy_install-2.7 algopy`
That's all! Here are the terminal output messages
```sudo easy_install-2.7 algopy
Searching for algopy
Best match: algopy 0.3.1
Processing algopy-0.3.1.zip
Running algopy-0.3.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-SvuPun/algopy-0.3.1/egg-dist-tmp-8pSEhA
.dev
warning: manifest_maker: MANIFEST.in, line 1: 'recursive-include' expects    ...

zip_safe flag not set; analyzing archive contents...
algopy.__init__: module references __file__
algopy.tracer.tests.environment: module references __file__
Adding algopy 0.3.1 to easy-install.pth file

Installed /usr/local/lib/python2.7/dist-packages/algopy-0.3.1-py2.7.egg
Processing dependencies for algopy
Finished processing dependencies for algopy
```
Actually easy_install has three versions in my machine, namely easy_install easy_install-2.6 easy_install-2.7 I got the following error for instance by typing easy_install only:
```sudo easy_install algopy
Traceback (most recent call last):
File "/usr/local/bin/easy_install", line 5, in
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2676, in
parse_requirements(__requires__), Environment()
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 552, in resolve
raise DistributionNotFound(req)
```
It is much better to install easy_install to your system than downloading the tarred package from the homepage and unpacking and installing it via python setup.py install

#### Trying out an example

```import numpy, algopy
from algopy import UTPM, exp

def eval_f(x):
""" some function """
return x[0]*x[1]*x[2] + exp(x[0])*x[1]

# forward mode without building the computational graph
# -----------------------------------------------------
x = UTPM.init_jacobian([3,5,7])
y = eval_f(x)
algopy_jacobian = UTPM.extract_jacobian(y)
print 'jacobian = ',algopy_jacobian

# reverse mode using a computational graph
# ----------------------------------------

# STEP 1: trace the function evaluation
cg = algopy.CGraph()
x = algopy.Function([1,2,3])
y = eval_f(x)
cg.trace_off()
cg.independentFunctionList = [x]
cg.dependentFunctionList = [y]

# STEP 2: use the computational graph to evaluate derivatives
print 'Jacobian =', cg.jacobian([[3.,5,7]])
print 'Hessian =', cg.hessian([[3.,5.,7.]])
print 'Hessian vector product =', cg.hess_vec([[3.,5.,7.]],[[4,5,6]])
```
When the above is executed in my laptop, it prints to the console or terminal the following:
```.
>>> import numpy, algopy
>>> from algopy import UTPM, exp
>>>
>>> def eval_f(x):
...     """ some function """
...     return x[0]*x[1]*x[2] + exp(x[0])*x[1]
...
>>> # forward mode without building the computational graph
... # -----------------------------------------------------
... x = UTPM.init_jacobian([3,5,7])
>>> y = eval_f(x)
>>> algopy_jacobian = UTPM.extract_jacobian(y)
>>> print 'jacobian = ',algopy_jacobian
jacobian =  [ 135.42768462   41.08553692   15.        ]
>>>
>>> # reverse mode using a computational graph
... # ----------------------------------------
...
>>> # STEP 1: trace the function evaluation
... cg = algopy.CGraph()
>>> x = algopy.Function([1,2,3])
>>> y = eval_f(x)
>>> cg.trace_off()

>>> cg.independentFunctionList = [x]
>>> cg.dependentFunctionList = [y]
>>>
>>> # STEP 2: use the computational graph to evaluate derivatives
gradient = [array([ 135.42768462,   41.08553692,   15.        ])]
>>> print 'Jacobian =', cg.jacobian([[3.,5,7]])
Jacobian =
Traceback (most recent call last):
File "", line 1, in
File "/usr/local/lib/python2.7/dist-packages/algopy-0.3.1-py2.7.egg/algopy/tracer/tracer.py", line 262, in jacobian
self.pushforward(utpm_x_list)
File "/usr/local/lib/python2.7/dist-packages/algopy-0.3.1-py2.7.egg/algopy/tracer/tracer.py", line 103, in pushforward
f.__class__.pushforward(f.func, f.args, Fout = f)
File "/usr/local/lib/python2.7/dist-packages/algopy-0.3.1-py2.7.egg/algopy/tracer/tracer.py", line 637, in pushforward
out  = func(*args)
File "/usr/local/lib/python2.7/dist-packages/algopy-0.3.1-py2.7.egg/algopy/utpm/utpm.py", line 73, in __getitem__
tmp = self.data.__getitem__((slice(None),slice(None)) + tuple(sl))
IndexError: invalid index
>>> print 'Hessian =', cg.hessian([[3.,5.,7.]])
Hessian =
Traceback (most recent call last):
File "", line 1, in
File "/usr/local/lib/python2.7/dist-packages/algopy-0.3.1-py2.7.egg/algopy/tracer/tracer.py", line 349, in hessian
self.pushforward(utpm_x_list)
File "/usr/local/lib/python2.7/dist-packages/algopy-0.3.1-py2.7.egg/algopy/tracer/tracer.py", line 103, in pushforward
f.__class__.pushforward(f.func, f.args, Fout = f)
File "/usr/local/lib/python2.7/dist-packages/algopy-0.3.1-py2.7.egg/algopy/tracer/tracer.py", line 637, in pushforward
out  = func(*args)
File "/usr/local/lib/python2.7/dist-packages/algopy-0.3.1-py2.7.egg/algopy/utpm/utpm.py", line 73, in __getitem__
tmp = self.data.__getitem__((slice(None),slice(None)) + tuple(sl))
IndexError: invalid index
>>> print 'Hessian vector product =', cg.hess_vec([[3.,5.,7.]],[
```

Ouch, we have fatal errors! We do have the latest version of AlgoPy (0.3.1). But you may be luckier than me.

We are a little sorry for this post. But we will come back soon, we need to visit the mailing list or the algopy group (if there is). We will come back soon!

## Wednesday, March 7, 2012

### Its about time Python get serious with Statistics! Introducing Scikits.statsmodels (Linux Ubuntu)), Part 1

You can of course perform powerful statistical routines using Python, Rpy module and R. But greater joy is now coming to pure Python users with the Scikits.statsmodels package.

• Install the scikits.statsmodels module
• Be sure that your versions of scipy and numpy are the latest or later than the following:
```Python >= 2.5
NumPy >= 1.4.0
SciPy >= 0.7
```
You can install the latest versions by typing sudo apt-get install python python-setuptools python-numpy python-scipy
• In a terminal type wget -ct 0 http://pypi.python.org/packages/source/s/scikits.statsmodels/scikits.statsmodels-0.3.1.tar.gz#md5=1f55b53d161544b95ca2709c9731c00c Untar it,then descend to the directory.
```tar -xvvf scikits.statsmodels-0.3.1.tar.gz
sudo python setup.py install
```
• Test the installed module
• Lets try this from ipython(Again, do a sudo apt-get install ipython if it is not yet installed.
```import scikits.statsmodels.api as sm
help(sm)
```
The following will be printed:
`Help on module scikits.statsmodels.api in scikits.statsmodels:  NAME     scikits.statsmodels.api - Statistical models  FILE     /home/toto/testsoftware/scikits.statsmodels-0.3.1/scikits/statsmodels/api.py  DESCRIPTION      - standard `regression` models            - `GLS` (generalized least squares regression)       - `OLS` (ordinary least square regression)       - `WLS` (weighted least square regression)       - `GLASAR` (GLS with autoregressive errors model)            - `GLM` (generalized linear models)       - robust statistical models            - `RLM` (robust linear models using M estimators)       - `robust.norms` estimates       - `robust.scale` estimates (MAD, Huber's proposal 2).       - sandbox models       - `mixed` effects models       - `gam` (generalized additive model `
What is available by importing sm? Type dir(sm) and the following functions or attibutes will be listed:
```In [17]: dir(sm)
Out[17]:
['GLM',
'GLS',
'GLSAR',
'Logit',
'MNLogit',
'OLS',
'Poisson',
'Probit',
'RLM',
'WLS',
'__builtins__',
'__doc__',
'__file__',
'__name__',
'__package__',
'categorical',
'datasets',
'families',
'iolib',
'nonparametric',
'regression',
'robust',
'test',
'tools',
'tsa',
'version']
```
• The regression example from the homepage
• It is time to show what statsmodels can do. Here is a regression example from the home page: http://statsmodels.sourceforge.net/
```import numpy as np
import scikits.statsmodels.api as sm

# get data
nsample = 100
x = np.linspace(0,10, 100)
beta = np.array([1, 0.1, 10])
y = np.dot(X, beta) + np.random.normal(size=nsample)

# run the regression
results = sm.OLS(y, X).fit()

# look at the results
print results.summary()
```
Here is the results by running the above published example:
```     Summary of Regression Results
=======================================
| Dependent Variable:            ['y']|
| Model:                           OLS|
| Method:                Least Squares|
| Date:               Wed, 07 Mar 2012|
| Time:                       17:58:02|
| # obs:                         100.0|
| Df residuals:                   97.0|
| Df model:                        2.0|
==============================================================================
|                   coefficient     std. error    t-statistic          prob. |
------------------------------------------------------------------------------
| x1                      1.060         0.1353         7.8344         0.0000 |
| x2                    0.09398        0.01309         7.1802         0.0000 |
| const                   9.848         0.2927        33.6490         0.0000 |
==============================================================================
|                          Models stats                      Residual stats  |
------------------------------------------------------------------------------
| R-squared:                     0.9729   Durbin-Watson:              1.803  |
| Adjusted R-squared:            0.9724   Omnibus:                    3.108  |
| F-statistic:                    1742.   Prob(Omnibus):             0.2114  |
| Prob (F-statistic):         9.776e-77   JB:                         2.791  |
| Log likelihood:                -139.9   Prob(JB):                  0.2477  |
| AIC criterion:                  285.8   Skew:                     -0.4176  |
| BIC criterion:                  293.6   Kurtosis:                   2.927  |
------------------------------------------------------------------------------

```
So far so good. We hope that the statsmodels module be more robust, easier to use and inspiring enough for Python users users to contribute code!

## Saturday, February 18, 2012

### Installing the latest free 3.5rc3 libreoffice productivity tools

The problem with the built-in package system in Ubuntu is that the softwares in the repositories may be one or two or even three versions behind! My old version of libreoffice was at 3.4. It is now at Version 3.5. But the disadvantage of manually installing the latest version is that there will be NO automatic updates!

```cd desktop-intergration