I had a problem determining the gradient of the ordinal logistic model and I could not find online references on the formula for the gradient. You see it takes around 6 seconds to compute the parameters of model using the BFGS (Broyden, Fletcher, Goldfarb, Shanno) optimizer.
I have heard of ADOL-C before for automatic differentiation, in fact I was browsing the second hand Book Sale and I found a reference book with Java-Forte programs on a CD. That turned me off, since Forte is relative old IDE and I decided NOT to buy it. So lets try an automatic differentiation package for Python.
| march 24, 2012 |
Oneiric Ocelot) on my laptop which serves as my main computer. Try to see the version of your installed default Python by issuing python --version. I see that I have 2.7.2+ Now try this: sudo easy_install-2.7 algopy That's all! Here are the terminal output messages sudo easy_install-2.7 algopy
Searching for algopy
Reading http://pypi.python.org/simple/algopy/
Reading http://packages.python.org/algopy
Reading http://www.github.com/b45ch1/algopy
Best match: algopy 0.3.1
Downloading http://pypi.python.org/packages/source/a/algopy/algopy-0.3.1.zip#md5=84edf4722e0f67ae254d663906893ff5
Processing algopy-0.3.1.zip
Running algopy-0.3.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-SvuPun/algopy-0.3.1/egg-dist-tmp-8pSEhA
.dev
warning: manifest_maker: MANIFEST.in, line 1: 'recursive-include' expects ...
zip_safe flag not set; analyzing archive contents...
algopy.__init__: module references __file__
algopy.tracer.tests.environment: module references __file__
Adding algopy 0.3.1 to easy-install.pth file
Installed /usr/local/lib/python2.7/dist-packages/algopy-0.3.1-py2.7.egg
Processing dependencies for algopy
Finished processing dependencies for algopy
Actually easy_install has three versions in my machine, namely easy_install easy_install-2.6 easy_install-2.7 I got the following error for instance by typing easy_install only: sudo easy_install algopy
[sudo] password for toto:
Traceback (most recent call last):
File "/usr/local/bin/easy_install", line 5, in
from pkg_resources import load_entry_point
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2676, in
parse_requirements(__requires__), Environment()
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 552, in resolve
raise DistributionNotFound(req)
It is much better to install easy_install to your system than downloading the tarred package from the homepage and unpacking and installing it via python setup.py install Trying out an exampleimport numpy, algopy
from algopy import UTPM, exp
def eval_f(x):
""" some function """
return x[0]*x[1]*x[2] + exp(x[0])*x[1]
# forward mode without building the computational graph
# -----------------------------------------------------
x = UTPM.init_jacobian([3,5,7])
y = eval_f(x)
algopy_jacobian = UTPM.extract_jacobian(y)
print 'jacobian = ',algopy_jacobian
# reverse mode using a computational graph
# ----------------------------------------
# STEP 1: trace the function evaluation
cg = algopy.CGraph()
x = algopy.Function([1,2,3])
y = eval_f(x)
cg.trace_off()
cg.independentFunctionList = [x]
cg.dependentFunctionList = [y]
# STEP 2: use the computational graph to evaluate derivatives
print 'gradient =', cg.gradient([[3.,5,7]])
print 'Jacobian =', cg.jacobian([[3.,5,7]])
print 'Hessian =', cg.hessian([[3.,5.,7.]])
print 'Hessian vector product =', cg.hess_vec([[3.,5.,7.]],[[4,5,6]])
When the above is executed in my laptop, it prints to the console or terminal the following: .
>>> import numpy, algopy
>>> from algopy import UTPM, exp
>>>
>>> def eval_f(x):
... """ some function """
... return x[0]*x[1]*x[2] + exp(x[0])*x[1]
...
>>> # forward mode without building the computational graph
... # -----------------------------------------------------
... x = UTPM.init_jacobian([3,5,7])
>>> y = eval_f(x)
>>> algopy_jacobian = UTPM.extract_jacobian(y)
>>> print 'jacobian = ',algopy_jacobian
jacobian = [ 135.42768462 41.08553692 15. ]
>>>
>>> # reverse mode using a computational graph
... # ----------------------------------------
...
>>> # STEP 1: trace the function evaluation
... cg = algopy.CGraph()
>>> x = algopy.Function([1,2,3])
>>> y = eval_f(x)
>>> cg.trace_off()
>>> cg.independentFunctionList = [x]
>>> cg.dependentFunctionList = [y]
>>>
>>> # STEP 2: use the computational graph to evaluate derivatives
... print 'gradient =', cg.gradient([[3.,5,7]])
gradient = [array([ 135.42768462, 41.08553692, 15. ])]
>>> print 'Jacobian =', cg.jacobian([[3.,5,7]])
Jacobian =
Traceback (most recent call last):
File "", line 1, in
File "/usr/local/lib/python2.7/dist-packages/algopy-0.3.1-py2.7.egg/algopy/tracer/tracer.py", line 262, in jacobian
self.pushforward(utpm_x_list)
File "/usr/local/lib/python2.7/dist-packages/algopy-0.3.1-py2.7.egg/algopy/tracer/tracer.py", line 103, in pushforward
f.__class__.pushforward(f.func, f.args, Fout = f)
File "/usr/local/lib/python2.7/dist-packages/algopy-0.3.1-py2.7.egg/algopy/tracer/tracer.py", line 637, in pushforward
out = func(*args)
File "/usr/local/lib/python2.7/dist-packages/algopy-0.3.1-py2.7.egg/algopy/utpm/utpm.py", line 73, in __getitem__
tmp = self.data.__getitem__((slice(None),slice(None)) + tuple(sl))
IndexError: invalid index
>>> print 'Hessian =', cg.hessian([[3.,5.,7.]])
Hessian =
Traceback (most recent call last):
File "", line 1, in
File "/usr/local/lib/python2.7/dist-packages/algopy-0.3.1-py2.7.egg/algopy/tracer/tracer.py", line 349, in hessian
self.pushforward(utpm_x_list)
File "/usr/local/lib/python2.7/dist-packages/algopy-0.3.1-py2.7.egg/algopy/tracer/tracer.py", line 103, in pushforward
f.__class__.pushforward(f.func, f.args, Fout = f)
File "/usr/local/lib/python2.7/dist-packages/algopy-0.3.1-py2.7.egg/algopy/tracer/tracer.py", line 637, in pushforward
out = func(*args)
File "/usr/local/lib/python2.7/dist-packages/algopy-0.3.1-py2.7.egg/algopy/utpm/utpm.py", line 73, in __getitem__
tmp = self.data.__getitem__((slice(None),slice(None)) + tuple(sl))
IndexError: invalid index
>>> print 'Hessian vector product =', cg.hess_vec([[3.,5.,7.]],[
Ouch, we have fatal errors! We do have the latest version of AlgoPy (0.3.1). But you may be luckier than me.
We are a little sorry for this post. But we will come back soon, we need to visit the mailing list or the algopy group (if there is). We will come back soon!
You can of course perform powerful statistical routines using Python, Rpy module and R. But greater joy is now coming to pure Python users with the Scikits.statsmodels package.
- Install the scikits.statsmodels module
Be sure that your versions of scipy and numpy are the latest or later than the following:
Python >= 2.5
NumPy >= 1.4.0
SciPy >= 0.7
You can install the latest versions by typing sudo apt-get install python python-setuptools python-numpy python-scipy
Download the software scikits.statsmodels module
In a terminal type wget -ct 0 http://pypi.python.org/packages/source/s/scikits.statsmodels/scikits.statsmodels-0.3.1.tar.gz#md5=1f55b53d161544b95ca2709c9731c00c Untar it,then descend to the directory. tar -xvvf scikits.statsmodels-0.3.1.tar.gz
sudo python setup.py install
Test the installed module
Lets try this from ipython(Again, do a sudo apt-get install ipython if it is not yet installed. import scikits.statsmodels.api as sm
help(sm)
The following will be printed:
Help on module scikits.statsmodels.api in scikits.statsmodels: NAME scikits.statsmodels.api - Statistical models FILE /home/toto/testsoftware/scikits.statsmodels-0.3.1/scikits/statsmodels/api.py DESCRIPTION - standard `regression` models - `GLS` (generalized least squares regression) - `OLS` (ordinary least square regression) - `WLS` (weighted least square regression) - `GLASAR` (GLS with autoregressive errors model) - `GLM` (generalized linear models) - robust statistical models - `RLM` (robust linear models using M estimators) - `robust.norms` estimates - `robust.scale` estimates (MAD, Huber's proposal 2). - sandbox models - `mixed` effects models - `gam` (generalized additive model
What is available by importing sm? Type dir(sm) and the following functions or attibutes will be listed:
In [17]: dir(sm)
Out[17]:
['GLM',
'GLS',
'GLSAR',
'Logit',
'MNLogit',
'OLS',
'Poisson',
'Probit',
'RLM',
'WLS',
'__builtins__',
'__doc__',
'__file__',
'__name__',
'__package__',
'add_constant',
'categorical',
'datasets',
'families',
'iolib',
'nonparametric',
'regression',
'robust',
'test',
'tools',
'tsa',
'version']
Additional resources
Home page of pystatsmodels: http://statsmodels.sourceforge.net/ Subscribe to the Google group: http://groups.google.com/group/pystatsmodels
The regression example from the homepage
It is time to show what statsmodels can do. Here is a regression example from the home page: http://statsmodels.sourceforge.net/
import numpy as np
import scikits.statsmodels.api as sm
# get data
nsample = 100
x = np.linspace(0,10, 100)
X = sm.add_constant(np.column_stack((x, x**2)))
beta = np.array([1, 0.1, 10])
y = np.dot(X, beta) + np.random.normal(size=nsample)
# run the regression
results = sm.OLS(y, X).fit()
# look at the results
print results.summary()
Here is the results by running the above published example:
Summary of Regression Results
=======================================
| Dependent Variable: ['y']|
| Model: OLS|
| Method: Least Squares|
| Date: Wed, 07 Mar 2012|
| Time: 17:58:02|
| # obs: 100.0|
| Df residuals: 97.0|
| Df model: 2.0|
==============================================================================
| coefficient std. error t-statistic prob. |
------------------------------------------------------------------------------
| x1 1.060 0.1353 7.8344 0.0000 |
| x2 0.09398 0.01309 7.1802 0.0000 |
| const 9.848 0.2927 33.6490 0.0000 |
==============================================================================
| Models stats Residual stats |
------------------------------------------------------------------------------
| R-squared: 0.9729 Durbin-Watson: 1.803 |
| Adjusted R-squared: 0.9724 Omnibus: 3.108 |
| F-statistic: 1742. Prob(Omnibus): 0.2114 |
| Prob (F-statistic): 9.776e-77 JB: 2.791 |
| Log likelihood: -139.9 Prob(JB): 0.2477 |
| AIC criterion: 285.8 Skew: -0.4176 |
| BIC criterion: 293.6 Kurtosis: 2.927 |
------------------------------------------------------------------------------
So far so good. We hope that the statsmodels module be more robust, easier to use and inspiring enough for Python users users to contribute code!
The problem with the built-in package system in Ubuntu is that the softwares in the repositories may be one or two or even three versions behind! My old version of libreoffice was at 3.4. It is now at Version 3.5. But the disadvantage of manually installing the latest version is that there will be NO automatic updates!
- Download the compressed installer
Visit http://www.libreoffice.org/download/ and choose http://download.documentfoundation.org/libreoffice/stable/3.5.0/deb/x86_64/LibO_3.5.0_Linux_x86-64_install-deb_en-US.tar.gz for 64 bit systems. Of course, if your system is 32 bit, choose http://download.documentfoundation.org/libreoffice/stable/3.5.0/deb/x86/LibO_3.5.0_Linux_x86_install-deb_en-US.tar.gz
- Uncompressed the downloaded file
For Ubuntu, the file will be saved in ~/home/Downloads/ subdirectory. cd to that in the terminal and issue tar xzvvf LibO_3.5.0_Linux_x86-64_install-deb_en-US.tar.gz. This will create a subdirectory LibO_3.5.0rc3_Linux_x86-64_install-deb_en-US/. cd to that directory.
- Remove old traces of LibreOffice
Now remove any trace of old versions of LibreOffice. If you have installed the old version using synaptic or the software installer of Ubuntu, you can try sudo apt-get remove libreoffice. Then issue sudo apt-get purge. To ensure there are no remaining traces of the old installation, try using locate libreoffice. The directories containing old versions are usually /opt/libreoffice, /opt/libreoffice3.4, /var/libreoffice and some others.
- Install the downloaded file
Simpy type on the terminal sudo dpkg -i *.deb in the uncompressed directory.
- Install the desktop integration.
In the same decompressed directory issue cd desktop-intergration
sudo dpkg -i *.deb
Your new LibreOffice installation should now be able to run from the Destkop GUI Applications.
Congratulations on having a free alternative to Microsoft Windows Office Applications in Linux Operating sytem.
As a public univerity supported by people's taxes, the transparency of UC Berkeley in selecting Google over Microsoft to provide online email and calendaring services is very welcome. Here is a decision table showing the strengths and weaknesses of the two opposite solutions (one free, the other commercial).
Productivity Suite Comparison Matrix
What is nice about Sympy?
It aims to be a simple but full-featured, extensible computer algebra system and it is written in pure Python. There is no need for external libraries!
Here are some information extracted from the home page of sympy.
What are the available features in Sympy?
The current core of Sympy have the following capabilities:
- basic arithmetics *,/,+,-,**
- basic simplification (like a*b*b + 2*b*a*b -> 3*a*b**2)
- expansion (like (a+b)**2 -> a**2 + 2*a*b + b**2)
- functions (exp, ln, ...)
- complex numbers (like exp(I*x).expand(complex=True) -> cos(x)+I*sin(x))
- differentiation
- taylor (laurent) series
- substitution (like x -> ln(x), or sin -> cos)
- arbitrary precision integers, rationals and floats
- noncommutative symbols
- pattern matching
What are the avaialable modules aside from the core?
- more functions (sin, cos, tan, atan, asin, acos, factorial, zeta, legendre)
- limits (like limit(x*log(x), x, 0) -> 0)
- integration using extended Risch-Norman heuristic
- polynomials (division, gcd, square free decomposition, groebner bases, factorization)
- solvers (algebraic, difference and differential equations, and systems of equations)
- symbolic matrices (determinants, LU decomposition...)
- mpmath (multiprecision floating-point arithmetic)
- geometric algebra (GA)
- Pauli and Dirac algebra
- quantum physics
- geometry module
- plotting (2D and 3D)
- code generation (C, Fortran, LaTeX)
We wish that Sympy will become standard and integrate with the other package called Symbolic. It is a long time coming. This will not supplant the need for Maxima and will enable Python loving scientists and applied and research mathematicians. But I got the following error on installing Sympy in Oneiric:
sudo apt-get install sympy
[sudo] password for toto:
Reading package lists... Done
Building dependency tree Reading state information... Done
E: Unable to locate package sympy
toto@toto-Aspire-4520:~$ sudo apt-get install python-sympy
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
python-sympy 0 upgraded, 1 newly installed, 0 to remove and 2 not upgraded.
Need to get 1,694 kB of archives. After this operation, 9,490 kB of additional disk space will be used.
Get:1 http://ph.archive.ubuntu.com/ubuntu/ oneiric/universe python-sympy all 0.6.7-1.1 [1,694 kB] Fetched 1,694 kB in 19s (84.8 kB/s)
Selecting previously deselected package python-sympy. (Reading database ... 552034 files and directories currently installed.)
Unpacking python-sympy (from .../python-sympy_0.6.7-1.1_all.deb) ... Processing triggers for man-db ...
Processing triggers for menu ... Setting up python-sympy (0.6.7-1.1) ... Processing triggers for menu ...
Processing triggers for python-support ... Compiling /usr/lib/pymodules/python2.6/mpmath/libmp/exec_py3.py ... SyntaxError: ('invalid syntax', ('/usr/lib/pymodules/python2.6/mpmath/libmp/exec_py3.py', 1, 12, 'exec_ = exec\n')) Compiling /usr/lib/pymodules/python2.7/mpmath/libmp/exec_py3.py ... SyntaxError: ('invalid syntax', ('/usr/lib/pymodules/python2.7/mpmath/libmp/exec_py3.py', 1, 12, 'exec_ = exec\n')) Compiling /usr/lib/pymodules/python2.7/mpmath/tests/extratest_gamma.py ... SyntaxError: ('invalid syntax', ('/usr/lib/pymodules/python2.7/mpmath/tests/extratest_gamma.py', 50, 35, ' print("%s ok;" % name, end=\' \')\n')) Compiling /usr/lib/pymodules/python2.7/mpmath/tests/torture.py ... SyntaxError: ('invalid syntax', ('/usr/lib/pymodules/python2.7/mpmath/tests/torture.py', 80, 27, ' print(".", end=\' \')\n'))
Yet when I fired up ipython and imported Sympy, there were no errors. But lets spend more time with this package on our blog. Oh, I removed sympy and reinstalled the package using
sudo easy_install-2.7 sympy
Here are some simple examples.
>>> sympy.factor("(x^2 + 2*a*x + a^2)")
(a + x)**2
>>> sympy.diff("x^2*y", "x")
2*x*y
>>> sympy.diff("log(y)")
1/y
>>> sympy.simplify("(x-a)/(x-a)")
1
Of course we just cannot trust the results blindly. Integration is somewhat convoluted. Here is the help documentation for integrate:
integrate(expr, *args, **kwargs)
integrate(f, var, ...)
Compute definite or indefinite integral of one or more variables
using Risch-Norman algorithm and table lookup. This procedure is
able to handle elementary algebraic and transcendental functions
and also a huge class of special functions, including Airy,
Bessel, Whittaker and Lambert.
var can be:
- a symbol -- indefinite integration
- a tuple (symbol, a, b) -- definite integration
Several variables can be specified, in which case the result is multiple
integration.
Also, if no var is specified at all, then the full anti-derivative of f is
returned. This is equivalent to integrating f over all its variables.
**Examples**
>>> from sympy import integrate, log
>>> from sympy.abc import a, x, y
>>> integrate(x*y, x)
x**2*y/2
>>> integrate(log(x), x)
x*log(x) - x
This is module worth having in your system. We hope that this will be more robust as time goes by.
|
