csatblogspotdotcom

Sunday, December 25, 2016

安装pip与Scrapy遇到的问题

安装pip与Scrapy(具体过程参考官网、与: http://cuiqingcai.com/912.html http://pip-cn.readthedocs.io/en/latest/installing.html https://raw.github.com/pypa/pip/master/contrib/get-pip.py(GitHub官网脚本) https://github.com/pypa/pip )后,发现root权限执行pip、scrapy正常,而普通用户报错: 执行python -m pip -V提示/usr/bin/python: No module named pip 执行scrapy提示: Traceback (most recent call last): File "/usr/local/bin/scrapy", line 7, in from scrapy.cmdline import execute ImportError: No module named scrapy.cmdline 执行python -c "import pip; print(pip.__version__)"提示: Traceback (most recent call last): File "", line 1, in ImportError: No module named pip 执行pip提示: Traceback (most recent call last): File "/usr/local/bin/pip", line 7, in from pip import main ImportError: No module named pip 而/usr/local/bin/pip的第七行是“from pip import main” 在自己的py程序中手动import pip也不行,于是查看路径,print sys.path显示: ['/home/****/py', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-x86_64-linux-gnu', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages', '/home/****/py/mymod'] 看了下,主要是/usr/lib/和/usr/local/lib/,pip和scrapy装在/usr/local/lib/,而这个目录下的文件和文件夹权限有些异常: alex@ubuntu:~$ ls -al /usr/local/lib/python2.7/dist-packages/ total 1176 drwxrwsr-x 56 root staff 4096 Dec 25 20:11 . drwxrwsr-x 4 root staff 4096 Dec 4 22:04 .. drwx--S--- 2 root staff 4096 Dec 4 22:17 attr drwx--S--- 2 root staff 4096 Dec 4 22:17 attrs-16.3.0.dist-info drwx--S--- 2 root staff 4096 Dec 4 22:09 certifi drwx--S--- 2 root staff 4096 Dec 4 22:09 certifi-2016.9.26.dist-info drwx--S--- 2 root staff 4096 Dec 4 21:58 cffi drwx--S--- 2 root staff 4096 Dec 4 21:58 cffi-1.9.1.dist-info -rwx--x--x 1 root staff 733736 Dec 4 21:58 _cffi_backend.so drwx--S--- 2 root staff 4096 Dec 4 22:17 constantly drwx--S--- 2 root staff 4096 Dec 4 22:17 constantly-15.1.0.dist-info drwx--S--- 4 root staff 4096 Dec 4 22:06 cryptography drwx--S--- 2 root staff 4096 Dec 4 22:06 cryptography-1.6-py2.7.egg-info drwx--S--- 2 root staff 4096 Dec 4 22:17 cssselect drwx--S--- 2 root staff 4096 Dec 4 22:17 cssselect-1.0.0.dist-info -rw------- 1 root staff 126 Dec 4 20:57 easy_install.py -rw------- 1 root staff 315 Dec 4 20:57 easy_install.pyc drwx--S--- 2 root staff 4096 Dec 25 19:48 enum drwx--S--- 2 root staff 4096 Dec 4 21:58 enum34-1.1.6.dist-info drwx--S--- 2 root staff 4096 Dec 4 21:58 idna drwx--S--- 2 root staff 4096 Dec 4 21:58 idna-2.1.dist-info drwx--S--- 3 root staff 4096 Dec 4 22:17 incremental drwx--S--- 2 root staff 4096 Dec 4 22:17 incremental-16.10.1.dist-info drwx--S--- 2 root staff 4096 Dec 4 21:58 ipaddress-1.0.17.dist-info -rw------- 1 root staff 80176 Dec 4 21:58 ipaddress.py -rw------- 1 root staff 75654 Dec 4 21:58 ipaddress.pyc drwx--S--- 2 root staff 4096 Dec 4 21:58 .libs_cffi_backend drwx--S--- 6 root staff 4096 Dec 4 21:23 lxml drwx--S--- 2 root staff 4096 Dec 4 21:23 lxml-3.6.4.dist-info drwx--S--- 3 root staff 4096 Dec 4 21:31 ndg drwx--S--- 2 root staff 4096 Dec 4 21:31 ndg_httpsclient-0.4.2-py2.7.egg-info -rw------- 1 root staff 346 Dec 4 21:31 ndg_httpsclient-0.4.2-py2.7-nspkg.pth drwx--S--- 2 root staff 4096 Dec 4 22:06 OpenSSL drwx--S--- 2 root staff 4096 Dec 4 22:17 parsel drwx--S--- 2 root staff 4096 Dec 4 22:17 parsel-1.1.0.dist-info drwx--S--- 10 root staff 4096 Dec 25 19:49 pip drwx--S--- 2 root staff 4096 Dec 4 20:57 pip-9.0.1.dist-info drwx--S--- 4 root staff 4096 Dec 4 20:57 pkg_resources drwx--S--- 5 root staff 4096 Dec 4 21:31 pyasn1 drwx--S--- 2 root staff 4096 Dec 4 21:31 pyasn1-0.1.9.dist-info drwx--S--- 2 root staff 4096 Dec 4 22:17 pyasn1_modules drwx--S--- 2 root staff 4096 Dec 4 22:17 pyasn1_modules-0.0.8.dist-info drwx--S--- 3 root staff 4096 Dec 4 21:58 pycparser drwx--S--- 2 root staff 4096 Dec 4 21:58 pycparser-2.17-py2.7.egg-info drwx--S--- 2 root staff 4096 Dec 4 22:17 pydispatch drwx--S--- 2 root staff 4096 Dec 4 22:17 PyDispatcher-2.0.5-py2.7.egg-info drwx--S--- 2 root staff 4096 Dec 4 22:06 pyOpenSSL-16.2.0.dist-info drwx--S--- 3 root staff 4096 Dec 4 22:17 queuelib drwx--S--- 2 root staff 4096 Dec 4 22:17 queuelib-1.4.2.dist-info drwx--S--- 20 root staff 4096 Dec 4 22:17 scrapy drwx--S--- 2 root staff 4096 Dec 4 22:17 Scrapy-1.2.1.dist-info drwx--S--- 2 root staff 4096 Dec 4 22:17 service_identity drwx--S--- 2 root staff 4096 Dec 4 22:17 service_identity-16.0.0.dist-info drwx--S--- 4 root staff 4096 Dec 4 20:57 setuptools drwx--S--- 2 root staff 4096 Dec 4 20:57 setuptools-30.1.0.dist-info drwx--S--- 2 root staff 4096 Dec 4 21:58 six-1.10.0.dist-info -rw------- 1 root staff 30098 Dec 4 21:58 six.py -rw------- 1 root staff 29545 Dec 4 21:58 six.pyc drwx--S--- 26 root staff 4096 Dec 4 22:17 twisted drwx--S--- 2 root staff 4096 Dec 4 22:17 Twisted-16.6.0-py2.7.egg-info drwx--S--- 2 root staff 4096 Dec 4 22:17 w3lib drwx--S--- 2 root staff 4096 Dec 4 22:17 w3lib-1.16.0.dist-info drwx--S--- 5 root staff 4096 Dec 4 20:57 wheel drwx--S--- 2 root staff 4096 Dec 4 20:57 wheel-0.29.0.dist-info alex@ubuntu:~$ ls -al /usr/local/lib/python2.7/dist-packages/pip ls: cannot open directory /usr/local/lib/python2.7/dist-packages/pip: Permission denied alex@ubuntu:~$ sudo ls -al /usr/local/lib/python2.7/dist-packages/pip [sudo] password for alex: total 420 drwx--S--- 10 root staff 4096 Dec 25 19:49 . drwxrwsr-x 56 root staff 4096 Dec 25 20:11 .. -rw------- 1 root staff 11910 Dec 4 20:57 basecommand.py -rw------- 1 root staff 8874 Dec 4 20:57 basecommand.pyc -rw------- 1 root staff 10465 Dec 4 20:57 baseparser.py -rw------- 1 root staff 10596 Dec 4 20:57 baseparser.pyc -rw------- 1 root staff 16474 Dec 4 20:57 cmdoptions.py -rw------- 1 root staff 15486 Dec 4 20:57 cmdoptions.pyc drwx--S--- 2 root staff 4096 Dec 4 20:57 commands drwx--S--- 2 root staff 4096 Dec 4 20:57 compat -rw------- 1 root staff 32171 Dec 4 20:57 download.py -rw------- 1 root staff 24968 Dec 4 20:57 download.pyc -rw------- 1 root staff 8121 Dec 4 20:57 exceptions.py -rw------- 1 root staff 12182 Dec 4 20:57 exceptions.pyc -rw------- 1 root staff 39950 Dec 4 20:57 index.py -rw------- 1 root staff 34576 Dec 4 20:57 index.pyc -rw------- 1 root staff 11348 Dec 4 20:57 __init__.py -rw------- 1 root staff 9163 Dec 4 20:57 __init__.pyc -rw------- 1 root staff 5626 Dec 4 20:57 locations.py -rw------- 1 root staff 4734 Dec 4 20:57 locations.pyc -rw------- 1 root staff 584 Dec 4 20:57 __main__.py -rw------- 1 root staff 477 Dec 4 20:57 __main__.pyc drwx--S--- 2 root staff 4096 Dec 4 20:57 models drwx--S--- 2 root staff 4096 Dec 4 20:57 operations -rw------- 1 root staff 10980 Dec 4 20:57 pep425tags.py -rw------- 1 root staff 9431 Dec 4 20:57 pep425tags.pyc drwx--S--- 2 root staff 4096 Dec 4 20:57 req -rw------- 1 root staff 156 Dec 4 20:57 status_codes.py -rw------- 1 root staff 386 Dec 4 20:57 status_codes.pyc drwx--S--- 2 root staff 4096 Dec 4 20:57 utils drwx--S--- 2 root staff 4096 Dec 4 20:57 vcs drwx--S--- 12 root staff 4096 Dec 4 20:57 _vendor -rw------- 1 root staff 32010 Dec 4 20:57 wheel.py -rw------- 1 root staff 25061 Dec 4 20:57 wheel.pyc 在自己的py程序中手动import enum(与pip都在/usr/local/lib/)也不行,于是怀疑整个/usr/local/lib/python2.7/dist-packages/目录都有问题,查了下,参考: https://groups.google.com/forum/#!topic/comp.lang.python/CfBiVvvFOdY 执行“chmod -R a+rX 目录**”解决(-R递归,a表示u用户g组o其他 这些所有角色,X和可执行权限x相关但不是所有都加: execute/search only if the file is a directory or already has execute permission for some user (X)),执行后: alex@ubuntu:~/py$ ls -al /usr/local/lib/python2.7/dist-packages total 1176 drwxrwsr-x 56 root staff 4096 Dec 25 20:11 . drwxrwsr-x 4 root staff 4096 Dec 4 22:04 .. drwxr-sr-x 2 root staff 4096 Dec 4 22:17 attr drwxr-sr-x 2 root staff 4096 Dec 4 22:17 attrs-16.3.0.dist-info drwxr-sr-x 2 root staff 4096 Dec 4 22:09 certifi drwxr-sr-x 2 root staff 4096 Dec 4 22:09 certifi-2016.9.26.dist-info drwxr-sr-x 2 root staff 4096 Dec 4 21:58 cffi drwxr-sr-x 2 root staff 4096 Dec 4 21:58 cffi-1.9.1.dist-info -rwxr-xr-x 1 root staff 733736 Dec 4 21:58 _cffi_backend.so drwxr-sr-x 2 root staff 4096 Dec 4 22:17 constantly drwxr-sr-x 2 root staff 4096 Dec 4 22:17 constantly-15.1.0.dist-info drwxr-sr-x 4 root staff 4096 Dec 4 22:06 cryptography drwxr-sr-x 2 root staff 4096 Dec 4 22:06 cryptography-1.6-py2.7.egg-info drwxr-sr-x 2 root staff 4096 Dec 4 22:17 cssselect drwxr-sr-x 2 root staff 4096 Dec 4 22:17 cssselect-1.0.0.dist-info -rw-r--r-- 1 root staff 126 Dec 4 20:57 easy_install.py -rw-r--r-- 1 root staff 315 Dec 4 20:57 easy_install.pyc drwxr-sr-x 2 root staff 4096 Dec 25 19:48 enum drwxr-sr-x 2 root staff 4096 Dec 4 21:58 enum34-1.1.6.dist-info drwxr-sr-x 2 root staff 4096 Dec 4 21:58 idna drwxr-sr-x 2 root staff 4096 Dec 4 21:58 idna-2.1.dist-info drwxr-sr-x 3 root staff 4096 Dec 4 22:17 incremental drwxr-sr-x 2 root staff 4096 Dec 4 22:17 incremental-16.10.1.dist-info drwxr-sr-x 2 root staff 4096 Dec 4 21:58 ipaddress-1.0.17.dist-info -rw-r--r-- 1 root staff 80176 Dec 4 21:58 ipaddress.py -rw-r--r-- 1 root staff 75654 Dec 4 21:58 ipaddress.pyc drwxr-sr-x 2 root staff 4096 Dec 4 21:58 .libs_cffi_backend drwxr-sr-x 6 root staff 4096 Dec 4 21:23 lxml drwxr-sr-x 2 root staff 4096 Dec 4 21:23 lxml-3.6.4.dist-info drwxr-sr-x 3 root staff 4096 Dec 4 21:31 ndg drwxr-sr-x 2 root staff 4096 Dec 4 21:31 ndg_httpsclient-0.4.2-py2.7.egg-info -rw-r--r-- 1 root staff 346 Dec 4 21:31 ndg_httpsclient-0.4.2-py2.7-nspkg.pth drwxr-sr-x 2 root staff 4096 Dec 4 22:06 OpenSSL drwxr-sr-x 2 root staff 4096 Dec 4 22:17 parsel drwxr-sr-x 2 root staff 4096 Dec 4 22:17 parsel-1.1.0.dist-info drwxr-sr-x 10 root staff 4096 Dec 25 19:49 pip drwxr-sr-x 2 root staff 4096 Dec 4 20:57 pip-9.0.1.dist-info drwxr-sr-x 4 root staff 4096 Dec 4 20:57 pkg_resources drwxr-sr-x 5 root staff 4096 Dec 4 21:31 pyasn1 drwxr-sr-x 2 root staff 4096 Dec 4 21:31 pyasn1-0.1.9.dist-info drwxr-sr-x 2 root staff 4096 Dec 4 22:17 pyasn1_modules drwxr-sr-x 2 root staff 4096 Dec 4 22:17 pyasn1_modules-0.0.8.dist-info drwxr-sr-x 3 root staff 4096 Dec 4 21:58 pycparser drwxr-sr-x 2 root staff 4096 Dec 4 21:58 pycparser-2.17-py2.7.egg-info drwxr-sr-x 2 root staff 4096 Dec 4 22:17 pydispatch drwxr-sr-x 2 root staff 4096 Dec 4 22:17 PyDispatcher-2.0.5-py2.7.egg-info drwxr-sr-x 2 root staff 4096 Dec 4 22:06 pyOpenSSL-16.2.0.dist-info drwxr-sr-x 3 root staff 4096 Dec 4 22:17 queuelib drwxr-sr-x 2 root staff 4096 Dec 4 22:17 queuelib-1.4.2.dist-info drwxr-sr-x 20 root staff 4096 Dec 4 22:17 scrapy drwxr-sr-x 2 root staff 4096 Dec 4 22:17 Scrapy-1.2.1.dist-info drwxr-sr-x 2 root staff 4096 Dec 4 22:17 service_identity drwxr-sr-x 2 root staff 4096 Dec 4 22:17 service_identity-16.0.0.dist-info drwxr-sr-x 4 root staff 4096 Dec 4 20:57 setuptools drwxr-sr-x 2 root staff 4096 Dec 4 20:57 setuptools-30.1.0.dist-info drwxr-sr-x 2 root staff 4096 Dec 4 21:58 six-1.10.0.dist-info -rw-r--r-- 1 root staff 30098 Dec 4 21:58 six.py -rw-r--r-- 1 root staff 29545 Dec 4 21:58 six.pyc drwxr-sr-x 26 root staff 4096 Dec 4 22:17 twisted drwxr-sr-x 2 root staff 4096 Dec 4 22:17 Twisted-16.6.0-py2.7.egg-info drwxr-sr-x 2 root staff 4096 Dec 4 22:17 w3lib drwxr-sr-x 2 root staff 4096 Dec 4 22:17 w3lib-1.16.0.dist-info drwxr-sr-x 5 root staff 4096 Dec 4 20:57 wheel drwxr-sr-x 2 root staff 4096 Dec 4 20:57 wheel-0.29.0.dist-info alex@ubuntu:~/py$ ls -al /usr/local/lib/python2.7/dist-packages/pip total 420 drwxr-sr-x 10 root staff 4096 Dec 25 19:49 . drwxrwsr-x 56 root staff 4096 Dec 25 20:11 .. -rw-r--r-- 1 root staff 11910 Dec 4 20:57 basecommand.py -rw-r--r-- 1 root staff 8874 Dec 4 20:57 basecommand.pyc -rw-r--r-- 1 root staff 10465 Dec 4 20:57 baseparser.py -rw-r--r-- 1 root staff 10596 Dec 4 20:57 baseparser.pyc -rw-r--r-- 1 root staff 16474 Dec 4 20:57 cmdoptions.py -rw-r--r-- 1 root staff 15486 Dec 4 20:57 cmdoptions.pyc drwxr-sr-x 2 root staff 4096 Dec 4 20:57 commands drwxr-sr-x 2 root staff 4096 Dec 4 20:57 compat -rw-r--r-- 1 root staff 32171 Dec 4 20:57 download.py -rw-r--r-- 1 root staff 24968 Dec 4 20:57 download.pyc -rw-r--r-- 1 root staff 8121 Dec 4 20:57 exceptions.py -rw-r--r-- 1 root staff 12182 Dec 4 20:57 exceptions.pyc -rw-r--r-- 1 root staff 39950 Dec 4 20:57 index.py -rw-r--r-- 1 root staff 34576 Dec 4 20:57 index.pyc -rw-r--r-- 1 root staff 11348 Dec 4 20:57 __init__.py -rw-r--r-- 1 root staff 9163 Dec 4 20:57 __init__.pyc -rw-r--r-- 1 root staff 5626 Dec 4 20:57 locations.py -rw-r--r-- 1 root staff 4734 Dec 4 20:57 locations.pyc -rw-r--r-- 1 root staff 584 Dec 4 20:57 __main__.py -rw-r--r-- 1 root staff 477 Dec 4 20:57 __main__.pyc drwxr-sr-x 2 root staff 4096 Dec 4 20:57 models drwxr-sr-x 2 root staff 4096 Dec 4 20:57 operations -rw-r--r-- 1 root staff 10980 Dec 4 20:57 pep425tags.py -rw-r--r-- 1 root staff 9431 Dec 4 20:57 pep425tags.pyc drwxr-sr-x 2 root staff 4096 Dec 4 20:57 req -rw-r--r-- 1 root staff 156 Dec 4 20:57 status_codes.py -rw-r--r-- 1 root staff 386 Dec 4 20:57 status_codes.pyc drwxr-sr-x 2 root staff 4096 Dec 4 20:57 utils drwxr-sr-x 2 root staff 4096 Dec 4 20:57 vcs drwxr-sr-x 12 root staff 4096 Dec 4 20:57 _vendor -rw-r--r-- 1 root staff 32010 Dec 4 20:57 wheel.py -rw-r--r-- 1 root staff 25061 Dec 4 20:57 wheel.pyc 普通用户可执行pip、scrapy,而且原有/usr/local/lib/python2.7/dist-packages 整个目录及子目录均可让普通用户访问了,解决了之前一直存在但一直没发现的问题 后记20170211: 1.虚拟机安装新系统Ubuntu 14.04.5,/usr/local/lib/python2.7/dist-packages 和 /usr/local/lib/python3.4/dist-packages 均为空,且属性为root:staff 2.安装pip与scrapy时会有一些小问题,安装好相关的依赖就可以了,例如sudo apt-get install gcc、g++、libssl-dev、libffi-dev、python-dev,以及pip install pyopenssl、pyasn1、ndg-httpsclient、lxml、twisted 3.此次安装又遇到了另一个问题,运行scrapy提示: Traceback (most recent call last): File "/usr/local/bin/scrapy", line 7, in from scrapy.cmdline import execute File "/usr/local/lib/python2.7/dist-packages/scrapy/cmdline.py", line 9, in from scrapy.crawler import CrawlerProcess File "/usr/local/lib/python2.7/dist-packages/scrapy/crawler.py", line 7, in from twisted.internet import reactor, defer File "/usr/local/lib/python2.7/dist-packages/twisted/internet/reactor.py", line 38, in from twisted.internet import default File "/usr/local/lib/python2.7/dist-packages/twisted/internet/default.py", line 56, in install = _getInstallFunction(platform) File "/usr/local/lib/python2.7/dist-packages/twisted/internet/default.py", line 44, in _getInstallFunction from twisted.internet.epollreactor import install File "/usr/local/lib/python2.7/dist-packages/twisted/internet/epollreactor.py", line 24, in from twisted.internet import posixbase File "/usr/local/lib/python2.7/dist-packages/twisted/internet/posixbase.py", line 18, in from twisted.internet import error, udp, tcp File "/usr/local/lib/python2.7/dist-packages/twisted/internet/tcp.py", line 28, in from twisted.internet._newtls import ( File "/usr/local/lib/python2.7/dist-packages/twisted/internet/_newtls.py", line 21, in from twisted.protocols.tls import TLSMemoryBIOFactory, TLSMemoryBIOProtocol File "/usr/local/lib/python2.7/dist-packages/twisted/protocols/tls.py", line 63, in from twisted.internet._sslverify import _setAcceptableProtocols File "/usr/local/lib/python2.7/dist-packages/twisted/internet/_sslverify.py", line 38, in TLSVersion.TLSv1_1: SSL.OP_NO_TLSv1_1, AttributeError: 'module' object has no attribute 'OP_NO_TLSv1_1' 查了下很有可能是twisted版本与scrapy不匹配导致,原有Ubuntu上scrapy是1.2.1,twisted是16.6.0,而新Ubuntu上scrapy是1.3.1,twisted是17.1.0,于是 sudo pip uninstall twisted(卸载) sudo pip install twisted==16.6.0(指定版本安装) sudo pip show twisted(查询) sudo chmod -R a+rX /usr/local/lib/python2.7/dist-packages/(更改相关文件夹权限) 后再次运行scrapy就OK 后记20170226: 昨天和今天继续整了下Python2、Python3以及pip、pip3 Python2和Python3可以看做两个独立的软件,pip作为Python的模块可以安装在Python2(/usr/local/lib/python2.7/dist-packages/)也可以安装在Python3(/usr/local/lib/python3.4/dist-packages/),执行pip和pip3分别为Python何Python3安装模块,也会默认安装到对应的这两个不同路径中,而查询的时候,例如pip/pip3 show coursera-dl,pip和pip3显示的结果是一样的,不管是2还是3版本的模块都能显示。默认安装模块后普通用户没有权限,主要是/usr/local/lib/python2.7/dist-packages/、/usr/local/lib/python3.4/dist-packages/,以及/usr/local/bin/这几个路径里的文件没权限,使用这些命令增加权限:sudo chmod -R a+rx /usr/local/bin/ 、 sudo chmod -R a+rX 目录 ,sudo chmod -R a+x *py 、sudo chmod -R a+x *pyc ,注意小写x是所有全加可执行权限,大写X是如果owner有x或者是目录,则加x;而 sudo chmod -R a+x *py 只能对当前目录的*py增加x,如果下一级子目录,需要 sudo chmod -R a+x */*py ,再下一级则需 sudo chmod -R a+x */*/*py ,增加权限后就可以不加sudo执行/查看了。另外需要注意的是apt-get安装的python3-pip版本太旧没法用,需要用官方的get-pip.py或者源码编译。而且Ubuntu14.04自带Python3.4但没见有pip3需手动装。整好后 python -m pip 和 python3 -m pip 都可以用了,python -m pip list 和 python3 -m pip list 可以分别列出2和3版本安装的模块。 理一下过程: coursera-dl作为Python模块,建议装在Python3目录,而自己的系统默认Python2,直接把Python命令指向Python3就可以直接安装pip到Python3对应路径,但会影响全局,影响其他程序,更好的方法是下载get-pip.py(https://bootstrap.pypa.io/get-pip.py),修改第一行 #!/usr/bin/env python 为 #!/usr/bin/env python3 (加一个3),这样 sudo ./get-pip.py 即可作为Python3的模块安装到 /usr/local/lib/python3.4/dist-packages/ ,接下来使用pip3安装coursera-dl:sudo pip3 install coursera-dl (或者 sudo python3 -m pip install ** 指定Python3的模块pip去install),也是直接装到 /usr/local/lib/python3.4/dist-packages/ ,注意增加相关目录权限。接下来就可以使用Python3的模块coursera-dl了(命令在/usr/local/bin/coursera-dl,相关文件在/usr/local/lib/python3.4/dist-packages/)。 后记20170516: 这几天安装TensorFlow遇到一些问题,走了一大圈弯路,细节省略了,大致过程如下: Ubuntu14.04装了Python2.7.6想安装最新的2.7.13,apt-get源里面没有,于是从源码安装,哪知道安装的路径不一致,原有在/usr/bin/、/usr/lib/ 这些目录里,新装的在/usr/local/bin/、/usr/local/lib/里面,导致一系列问题,源码命令没有卸载命令,只能手动一点点删,删完再重新试着安装2.7.6,重新pip: 起因是想装TensorFlow,因为主用Python2.7(2.7.6,虽然也装了Python3),所以想把2.7.6升到最新的2.7.13,以为这样可以避免Python的SNIMissingWarning和InsecurePlatformWarning告警(事实上怎么折腾还是没法避免)。但是apt源里面最新的就是2.7.6没法升,于是就下载了2.7.13源码编译安装,装好之后用pip安装TensorFlow,装好后import tensorflow告警: No module named tensorflow /usr/local/lib/python2.7/dist-packages/tensorflow 查看/usr/local/lib/python2.7/dist-packages/里面明明是有这个tensorflow文件夹啊,于是 print sys.path : ['/home/****', '/usr/local/lib/python27.zip', '/usr/local/lib/python2.7', '/usr/local/lib/python2.7/plat-linux2', '/usr/local/lib/python2.7/lib-tk', '/usr/local/lib/python2.7/lib-old', '/usr/local/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/site-packages'] (或者 #!/usr/bin/env python # -*- coding: utf-8 -*- import sys print '\n'.join(sys.path) 分行显示 而使用 #!/usr/bin/env python 与 #!/usr/local/bin/python 与 #!/usr/bin/python 与 #!/usr/bin/python3 的结果不尽相同,/usr/bin/ 里是原来装的python,/usr/local/bin/ 里是新的、源码安装的python。 ) sys.path和原来不一样,/usr/lib/ 变为了 /usr/local/lib/ 而且 dist-packages 变为了 site-packages ,这就是源码安装和apt-get安装造成的差异,怎么办? 有人建议装 virtualenv (参考:https://github.com/tensorflow/tensorflow/issues/647),这个 virtualenv 帮助管理路径,但我不想装,想自己掌握操作,于是试着添加路径到sys.path,不是在**.py里面临时sys.path.append,而是永久增加,参考: https://askubuntu.com/questions/798382/cant-import-installed-packages-in-python https://leemendelowitz.github.io/blog/how-does-python-find-packages.html https://docs.python.org/2/tutorial/modules.html (里面搜索 sys.path) https://docs.python.org/2/library/sys.html#sys.path https://docs.python.org/2/library/site.html#module-site http://www.tuicool.com/articles/EB3miqr http://www.jb51.net/article/42183.htm 选择在 site-packages 目录下加 ****.pth 文件的方法,创建 mypath.pth 里面写上目录,路径可以用相对路径,如果同一目录,直接文件名也可,添加之后 sys.path 确实有增加****.pth里的内容,现在sys.path的问题看起来解决了,但是import tensorflow仍然有问题: ImportError: Importing the multiarray numpy extension module failed. Most likely you are trying to import a failed build of numpy. If you're working with a numpy git repo, try `git clean -xdf` (removes all files not under version control). Otherwise reinstall numpy. 参考: https://github.com/tensorflow/tensorflow/issues/559 http://stackoverflow.com/questions/34051737/numpy-core-multiarray-failed-to-import http://stackoverflow.com/questions/20518632/importerror-numpy-core-multiarray-failed-to-import 还是版本冲突的问题,一下也找不出哪儿装了两个版本的numpy了,于是考虑卸载源码安装的Python2.7.13,但是源码目录里面没有卸载选项,只能手动删除: /usr/local/bin/python* /usr/local/bin/pydoc2 /usr/local/lib/python2 /usr/local/include/python2 /usr/local/lib/pkgconfig/python2.pc /usr/local/lib/libpython2 删除和以上类似的路径和文件后,/usr/local/bin/ 里面仍有一些文件,看了下有些是pip安装的Python包,例如pillow(PIL),产生的,于是直接删掉,找了会也没法100%确认干净,算了(期间误删了一些lib、pip文件,只好从另一个系统里复制过来一遍) 接下来,恢复安装Python2.7.6(其实原本就有,现在是重新安装、修复下,以免缺少文件之类的),这之前要添加第三方apt源(ppa),参考: https://www.zhihu.com/question/37021474?sort=created https://launchpad.net/ubuntu/+ppas?name_filter=python https://launchpad.net/~fkrull/+archive/ubuntu/deadsnakes https://askubuntu.com/questions/865554/how-do-i-install-python-3-6-using-apt-get 执行: sudo add-apt-repository ppa:fkrull/deadsnakes sudo apt-get update 再执行: sudo apt-get install --reinstall python2.7 sudo apt-get install --only-upgrade python2.7 sudo apt-get -f install python2.7 然后再apt-get装Python3.6,再从 https://pip.pypa.io/en/stable/installing/ 下载get-pip.py安装pip: python2 get-pip.py,下载 https://bootstrap.pypa.io/3.4****(try3.2、3.3/3.4、etc)/get-pip.py 来安装pip3: python3 get-pip.py(将get-pip.py首行改为:#!/usr/bin/env python3,注意写明python3而不是python),接下来又出现: File "get-pip.py", line 20061, in lsb_release ****的问题,查了下是lsb_release的问题,看到某网友 which lsb_release (查看lsb_release程序所在位置),于是 vi lsb_release 进去看了看,第一行是: #! /usr/bin/python3 -Es 修改Python3为python,再执行就OK了,看来还是安装Python3或者pip3时把 lsb_release 覆盖了,还是版本2和3的冲突。参考: https://github.com/pypa/setuptools/issues/937 https://discourse.mailinabox.email/t/unable-to-run-mailinabox-command/1840/11 (居然有网友卸载了重装来解决。。。。) 这个问题解决后接下来把pip和python指向2版本的可执行文件(通过强制修改软链接 ln -sf),但是每次pip之后还是需要修改 /usr/local/lib/python2.7/dist-packages 目录的权限很麻烦,查了下解决方法: http://stackoverflow.com/questions/23629898/problems-installing-matplotlib-in-linux http://caligari.treboada.net/2014/09/23/python-pip-and-the-staff-group/ https://askubuntu.com/questions/736738/i-tried-to-install-python-package-then-some-error-is-showing-like-exeption https://github.com/sindresorhus/weechat-notification-center/issues/1 https://superuser.com/questions/272061/reload-a-linux-users-group-assignments-without-logging-out https://arkaitzj.wordpress.com/2010/03/08/linux-add-user-to-a-group-without-logout/ 有把当前用户加入staff组的、有修改目录所有者、修改目录权限的,有用apt-get代替pip安装的,还有使用 virtualenv 的,试了下加入staff组然后 newgrp staff 不logout就即时生效,不好使,还是用 sudo pip **** 然后 sudo chmod -R a+rX /usr/local/lib/python2.7/dist-packages/ 。 最终的 sys.path 是(分行显示): /home/**** /usr/lib/python2.7 /usr/lib/python2.7/plat-x86_64-linux-gnu /usr/lib/python2.7/lib-tk /usr/lib/python2.7/lib-old /usr/lib/python2.7/lib-dynload /usr/local/lib/python2.7/dist-packages /usr/lib/python2.7/dist-packages 而 from distutils.sysconfig import get_python_lib print(get_python_lib()) 是 /usr/lib/python2.7/dist-packages 最后参考 https://github.com/martin-gorner/tensorflow-mnist-tutorial/blob/master/INSTALL.txt 进行测试: git clone https://github.com/martin-gorner/tensorflow-mnist-tutorial.git cd tensorflow-mnist-tutorial python mnist_1.0_softmax.py 提示 ImportError: No module named _tkinter, please install the python-tk package 缺少tkinter库于是执行sudo apt-get install python-tk,OK 终于基于python2.7.6成功安装TensorFlow,同时保持了python2和python3并存,pip2和pip3并存 中间的小问题省略了,最后总结的经验就是: 尽量别源码安装python,可以用第三方源加入PPA后(可用ppa:fkrull/deadsnakes),用apt-get等工具安装新版本,没有最新的就算了。装好python再用pip安装package或lib,装完 sudo chmod -R a+rX /****/lib/****/dist-packages/ tensorflow的安装可参考: https://github.com/martin-gorner/tensorflow-mnist-tutorial/blob/master/INSTALL.txt sudo -H apt-get install git sudo -H pip install --upgrade matplotlib sudo -H pip install --upgrade tensorflow (缺少tkinter库需执行sudo apt-get install python-tk)

Labels: , , ,

0 Comments:

Post a Comment

Subscribe to Post Comments [Atom]

<< Home