[x3d-public] X3D python again
Brutzman, Donald (Don) (CIV)
brutzman at nps.edu
Sat Mar 16 21:29:23 PDT 2019
BLUF: finally got past multiple setup difficulties on Windows 10, several issues addressed. More work needed.
Providing this record to help us finish build reconciliations and potentially assist others encountering similar difficulties.
Thanks for Loren Peitso for helpful python advice. (Incidentally he recommends _not_ using pycharm as it contains some irregularities).
Pyjnius still croaking, see log below. Is Microsoft Visual C++ actually needed?
Jacoco still croaking. Now commented out, probably best to get everything else working first. If you want to replace with your local version in the lib directory, please go ahead.
Future: wondering, can't we avoid beautiful soup and lxml altogether by using stock Python libraries?
Some success! Target processScenes.x3d now creating filename.py and filename.future.py outputs, examples attached.
Target processScenes.python not working, perhaps because pyjnius build isn't yet successful? Excerpt:
C:\x3d-code\www.web3d.org\x3d\content\examples\Basic\CAD\CadGeometryExternPrototypes.future.py
processScene.python:
Traceback (most recent call last):
File "C:\x3d-code\www.web3d.org\x3d\content\examples\Basic\CAD\CadGeometryExternPrototypes.py", line 1, in <module>
import jnius_config
ModuleNotFoundError: No module named 'jnius_config'
Result: 1
All this work now checked into subversion. Good luck with next steps, we are getting closer.
Am hoping we can continue to simplify and simplify. This approach will make this implementation as portable as possible. Looking forward to continued spiral development and improvement, improving the language syntax for X3D in Python.
Troubleshooting log follows.
======================================================================
Following another troubled install of anaconda (navigator not working) on Windows 10, ripped everything out again.
Reinstalled stock python 3.7.2 again from python.org as localadmin for all users. also included in path and confirmed working:
$ python --version
Python 3.7.2
Then ran build.xml target 'configure' as discussed on last call. Progress: it ran, though quickly failing.
======================
build.xml
<target name="configuration" description="Test Python output">
<echo message="create imported files: autoclass.py fieldtypes.py mapToMethod.py"/>
<exec executable="python">
<arg value="autoclass.py"/>
</exec>
[...]
======================
ant -f C:\\x3d-code\\www.web3d.org\\x3d\\stylesheets\\java\\src\\python\\pyjnius configuration
configuration:
create imported files: autoclass.py fieldtypes.py mapToMethod.py
Traceback (most recent call last):
File "autoclass.py", line 1, in <module>
from bs4 import BeautifulSoup
ModuleNotFoundError: No module named 'bs4'
Result: 1 [...]
======================
looked up that error, search led to
How can I from bs4 import BeautifulSoup?
https://stackoverflow.com/questions/18831380/how-can-i-from-bs4-import-beautifulsoup
which recommended
pip install beautifulsoup4
which produced two errors, even when running bash with administrator permissions:
============================
$ pip install beautifulsoup4
Collecting beautifulsoup4
Downloading https://files.pythonhosted.org/packages/1d/5d/3260694a59df0ec52f8b4883f5d23b130bc237602a1411fa670eae12351e/beautifulsoup4-4.7.1-py3-none-any.whl (94kB)
Collecting soupsieve>=1.2 (from beautifulsoup4)
Downloading https://files.pythonhosted.org/packages/77/78/bca00cc9fa70bba1226ee70a42bf375c4e048fe69066a0d9b5e69bc2a79a/soupsieve-1.8-py2.py3-none-any.whl (88kB)
Installing collected packages: soupsieve, beautifulsoup4
Could not install packages due to an EnvironmentError: [WinError 5] Access is denied: 'c:\\program files (x86)\\python37-32\\Lib\\site-packages\\soupsieve'
Consider using the `--user` option or check the permissions.
You are using pip version 18.1, however version 19.0.3 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.
============================
Tried again, still failed. Then noted that path started with
c:\\program files (x86)\\python37-32
which is 32 bit. This means that the python download page wasn't able to distinguish 32--bit or 64-bit.
Ripped it all out again, cleaned up PATH (the uninstaller didn't do that). Rebooted.
Went back to python download page and specifically found/downloaded/installed 64-bit Python 3.7.2 for Windows. Great care required in selecting the correct installer, this is buried within multiple lists.
https://www.python.org/downloads/windows/
Installed with all options including PATH, excluding debugger options.
Ran bash shell as administrator to update pip and beautifulsoup4 as indicated above.
=====================================
$ python -m pip install --upgrade pip
Collecting pip
Using cached https://files.pythonhosted.org/packages/d8/f3/413bab4ff08e1fc4828dfc59996d721917df8e8583ea85385d51125dceff/pip-19.0.3-py2.py3-none-any.whl
Installing collected packages: pip
Found existing installation: pip 18.1
Uninstalling pip-18.1:
Successfully uninstalled pip-18.1
Successfully installed pip-19.0.3
=====================================
$ pip install beautifulsoup4
Collecting beautifulsoup4
Downloading https://files.pythonhosted.org/packages/1d/5d/3260694a59df0ec52f8b4883f5d23b130bc237602a1411fa670eae12351e/beautifulsoup4-4.7.1-py3-none-any.whl (94kB)
Collecting soupsieve>=1.2 (from beautifulsoup4)
Downloading https://files.pythonhosted.org/packages/77/78/bca00cc9fa70bba1226ee70a42bf375c4e048fe69066a0d9b5e69bc2a79a/soupsieve-1.8-py2.py3-none-any.whl (88kB)
Installing collected packages: soupsieve, beautifulsoup4
Successfully installed beautifulsoup4-4.7.1 soupsieve-1.8
=====================================
progress: now a different error
=====================================
Could not load definitions from resource org/jacoco/ant/antlib.xml. It could not be found.
ant -f C:\\x3d-code\\www.web3d.org\\x3d\\stylesheets\\java\\src\\python\\pyjnius configuration
configuration:
create imported files: autoclass.py fieldtypes.py mapToMethod.py
Traceback (most recent call last):
File "autoclass.py", line 50, in <module>
soup = BeautifulSoup(open("../../../../../../specifications/X3dUnifiedObjectModel-4.0.xml"), "xml")
File "C:\Program Files\Python37\lib\site-packages\bs4\__init__.py", line 196, in __init__
% ",".join(features))
bs4.FeatureNotFound: Couldn't find a tree builder with the features you requested: xml. Do you need to install a parser library?
Result: 1
=====================================
found some help:
bs4.FeatureNotFound: Couldn't find a tree builder with the features you requested: lxml. Do you need to install a parser library?
https://stackoverflow.com/questions/24398302/bs4-featurenotfound-couldnt-find-a-tree-builder-with-the-features-you-requeste
followed initial guidance there for supporting:
=====================================
$ pip install lxml
Collecting lxml
Using cached https://files.pythonhosted.org/packages/be/b9/d05cd1680537aa39d8f0546871a4e56d82a5fdc97ac4006c37076a94915a/lxml-4.3.2-cp37-cp37m-win_amd64.whl
Installing collected packages: lxml
Successfully installed lxml-4.3.2
=====================================
worked as administrator! logged out, logged in as user, continued to work. hooray at last.
=====================================
Could not load definitions from resource org/jacoco/ant/antlib.xml. It could not be found.
ant -f C:\\x3d-code\\www.web3d.org\\x3d\\stylesheets\\java\\src\\python\\pyjnius configuration
configuration:
create imported files: autoclass.py fieldtypes.py mapToMethod.py
BUILD SUCCESSFUL (total time: 2 seconds)
=====================================
So, some suggestions seem to be in order.
a. we should document prerequisites.
b. perhaps add another target that performs these installs, along with diagnostics.
So, I worked out Ant invocations of all of these pip installs and got them working.
While integrating that block, at long last noted that you had most of this already in target install.dependencies - ouch!
And so, renamed that target for proximity to configuration.prerequisites, and made it a dependency of the configuration target so that it is always run.
=====================================
localadmin at DESKTOP-2S09UKA /cygdrive/c/x3d-code/www.web3d.org/x3d/stylesheets/java/src/python/pyjnius
$ ant configuration.prerequisites
Buildfile: C:\x3d-code\www.web3d.org\x3d\stylesheets\java\src\python\pyjnius\build.xml
[taskdef] Could not load definitions from resource org/jacoco/ant/antlib.xml. It could not be found.
configuration.prerequisites:
[echo] ensure needed libraries are installed (initial installs likely to require administrator login)
[echo] node.js and npm are required
[echo] https://nodejs.org/en
[echo] https://docs.npmjs.com/cli/install
[exec] audited 34 packages in 1s
[exec] found 0 vulnerabilities
[exec]
[echo] Pyjnius is a Python library for accessing Java classes.
[echo] https://pyjnius.readthedocs.io/en/latest/installation.html
[echo] Alert: use Pyjnius page to confirm JAVA_HOME and PATH requirements
[exec] Error: Could not create the Java Virtual Machine.
[exec] Error: A fatal exception has occurred. Program will exit.
[exec] Unrecognized option: --version
[exec] Result: 1
[exec] Python 3.7.2
[echo] Alert: note administrator permissions are likely required
[exec] Requirement already up-to-date: pip in c:\program files\python37\lib\site-packages (19.0.3)
[exec] Collecting setuptools
[exec] Downloading https://files.pythonhosted.org/packages/d1/6a/4b2fcefd2ea0868810e92d519dacac1ddc64a2e53ba9e3422c3b62b378a6/setuptools-40.8.0-py2.py3-none-any.whl (575kB)
[exec] Installing collected packages: setuptools
[exec] Found existing installation: setuptools 40.6.2
[exec] Uninstalling setuptools-40.6.2:
[exec] Successfully uninstalled setuptools-40.6.2
[exec] Successfully installed setuptools-40.8.0
[exec] Collecting cython
[exec] Downloading https://files.pythonhosted.org/packages/14/48/43fd6c0b29450c57a4eff636384dd691bef05c49410edfeb2fe4e4e79ec2/Cython-0.29.6-cp37-cp37m-win_amd64.whl (1.7MB)
[exec] Installing collected packages: cython
[exec] Successfully installed cython-0.29.6
[exec] Collecting pyjnius
[exec] Downloading https://files.pythonhosted.org/packages/b6/57/c90acf31322e6417f06c90410dbfcb149633a6006b7efbf99dfebe177c1f/pyjnius-1.2.0.tar.gz
[exec] Collecting six>=1.7.0 (from pyjnius)
[exec] Using cached https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
[exec] Requirement already satisfied, skipping upgrade: cython in c:\program files\python37\lib\site-packages (from pyjnius) (0.29.6)
[exec] Installing collected packages: six, pyjnius
[exec] Running setup.py install for pyjnius: started
[exec] Running setup.py install for pyjnius: finished with status 'error'
[exec] Complete output from command "C:\Program Files\Python37\python.exe" -u -c "import setuptools, tokenize;__file__='C:\\cygwin64\\tmp\\pip-install-bb5fdimw\\pyjnius\\setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record C:\cygwin64\tmp\pip-record-rm9cetjj\install-record.txt --single-version-externally-managed --compile:
[exec] C:/Program Files/Java/jdk1.8.0_202
[exec] WARNING: Not able to assign machine() = AMD64 to a cpu value!
[exec] Using cpu = 'i386' instead!
[exec] JDK_HOME: C:/Program Files/Java/jdk1.8.0_202
[exec]
[exec] JRE_HOME: C:/Program Files/Java/jdk1.8.0_202\jre
[exec]
[exec] warning: [options] bootstrap class path not set in conjunction with -source 1.6
[exec] 1 warning
[exec] running install
[exec] running build
[exec] running build_py
[exec] creating build
[exec] creating build\lib.win-amd64-3.7
[exec] copying jnius_config.py -> build\lib.win-amd64-3.7
[exec] creating build\lib.win-amd64-3.7\jnius
[exec] copying jnius\reflect.py -> build\lib.win-amd64-3.7\jnius
[exec] copying jnius\signatures.py -> build\lib.win-amd64-3.7\jnius
[exec] copying jnius\__init__.py -> build\lib.win-amd64-3.7\jnius
[exec] creating build\lib.win-amd64-3.7\jnius\src
[exec] creating build\lib.win-amd64-3.7\jnius\src\org
[exec] creating build\lib.win-amd64-3.7\jnius\src\org\jnius
[exec] copying jnius\src\org\jnius\NativeInvocationHandler.class -> build\lib.win-amd64-3.7\jnius\src\org\jnius
[exec] copying jnius\src\org\jnius\NativeInvocationHandler.java -> build\lib.win-amd64-3.7\jnius\src\org\jnius
[exec] running build_ext
[exec] cythoning jnius\jnius.pyx to jnius\jnius.c
[exec] C:\Program Files\Python37\lib\site-packages\Cython\Compiler\Main.py:367: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: C:\cygwin64\tmp\pip-install-bb5fdimw\pyjnius\jnius\jnius.pyx
[exec] tree = Parsing.p_module(s, pxd, full_module_name)
[exec] building 'jnius' extension
[exec] error: Microsoft Visual C++ 14.0 is required. Get it with "Microsoft Visual C++ Build Tools": https://visualstudio.microsoft.com/downloads/
[exec]
[exec] ----------------------------------------
[exec] Command ""C:\Program Files\Python37\python.exe" -u -c "import setuptools, tokenize;__file__='C:\\cygwin64\\tmp\\pip-install-bb5fdimw\\pyjnius\\setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record C:\cygwin64\tmp\pip-record-rm9cetjj\install-record.txt --single-version-externally-managed --compile" failed with error code 1 in C:\cygwin64\tmp\pip-install-bb5fdimw\pyjnius\
[exec] Result: 1
[echo] bs4 Beautiful Soup is a Python library for pulling data out of HTML and XML files
[echo] https://www.crummy.com/software/BeautifulSoup/bs4/doc
[exec] Collecting bs4
[exec] Using cached https://files.pythonhosted.org/packages/10/ed/7e8b97591f6f456174139ec089c769f89a94a1a4025fe967691de971f314/bs4-0.0.1.tar.gz
[exec] Requirement already satisfied, skipping upgrade: beautifulsoup4 in c:\program files\python37\lib\site-packages (from bs4) (4.7.1)
[exec] Requirement already satisfied, skipping upgrade: soupsieve>=1.2 in c:\program files\python37\lib\site-packages (from beautifulsoup4->bs4) (1.8)
[exec] Installing collected packages: bs4
[exec] Running setup.py install for bs4: started
[exec] Running setup.py install for bs4: finished with status 'done'
[exec] Successfully installed bs4-0.0.1
[echo] lxml - XML and HTML with Python https://lxml.de
[exec] Requirement already up-to-date: lxml in c:\program files\python37\lib\site-packages (4.3.2)
BUILD SUCCESSFUL
Total time: 22 seconds
=====================================
(Finally) a nice feature: don't need admin permission if packages are up-to-date, and the output diagnostics provide confirmation.
c. Still have problems with:
"[exec] error: Microsoft Visual C++ 14.0 is required. Get it with "Microsoft Visual C++ Build Tools": https://visualstudio.microsoft.com/downloads/"
Is that really needed?
d. Still have problems with jacocoant.jar - can't seem to find it online 8(
Looked all around the jacoco pages without success, wondering how to get it (or build it).
Kept searching elsewhere and found it in the Maven mvnrepository:
https://mvnrepository.com/artifact/org.jacoco/org.jacoco.ant/
https://mvnrepository.com/artifact/org.jacoco/org.jacoco.ant/0.8.3
http://central.maven.org/maven2/org/jacoco/org.jacoco.ant/0.8.3/org.jacoco.ant-0.8.3.jar
Then put that in the project path with other .jars, i.e.
C:\x3d-code\www.web3d.org\x3d\stylesheets\java\lib
Then updated the build.xml file:
<taskdef uri="antlib:org.jacoco.ant" resource="org/jacoco/ant/antlib.xml">
<classpath path="../../../lib/org.jacoco.ant-0.8.3.jar"/>
<!-- <classpath path="~/Downloads/jacoco-0.8.3/lib/jacocoant.jar"/>-->
</taskdef>
However there is some kind of versionitis problem between your code and the 0.8.3 .jar, the following errors result:
C:\x3d-code\www.web3d.org\x3d\stylesheets\java\src\python\pyjnius\build.xml:87: The following error occurred while executing this line:
jar:file:/C:/x3d-code/www.web3d.org/x3d/stylesheets/java/lib/jacocoant.jar!/org/jacoco/ant/antlib.xml:17: taskdef A class needed by class org.jacoco.ant.ReportTask cannot be found: org/jacoco/core/analysis/ICoverageNode
using the classloader AntClassLoader[C:\x3d-code\www.web3d.org\x3d\stylesheets\java\lib\jacocoant.jar]
=====================================
On 2/17/2019 4:10 PM, John Carlson wrote:
> I am ready to talk about PythonSerializer.js or PythonPipeliningSerializer.js, but not X3dToPython.xslt. You will probably have to take the lead on that. If we’re just discussing the Python SAI, I have 2 solutions for that, both probably incomplete. But I haven’t recently run the pipelined programs under visual inspection to see if my last change fixed it or not. So we can catch up together.
>
> 2/19 at 2pm PM is fine with me.
>
> My friends have had the best luck with PyCharm. I suspect that Anaconda will work with PyCharm, but let’s not try it as long as you have a way of editing a python program. I will try to set up a jupyter here: http://jupyter.goexchange.de/hub/user/john/ contact me over phone for password, but I think we may need admin privileges to install pyjnius there (will have to talk to alex). It would be good to see any installed programs like R export X3DOM or X_ITE.
>
> I have removed x3d-public from the list to maintain some measure of confidentiality for alex’s machines.
>
> I have also incompletely added JACOCO to build.xml (needs work).
>
> John
>
> Sent from Mail <https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10
>
> *From: *Brutzman, Donald (Don) (CIV) <mailto:brutzman at nps.edu>
> *Sent: *Sunday, February 17, 2019 11:24 AM
> *To: *John Carlson <mailto:yottzumm at gmail.com>
> *Cc: *X3D Graphics public mailing list <mailto:x3d-public at web3d.org>
> *Subject: *X3D python again
>
> hi John.
>
> Even after recent retries, never did get python 3.7.2 stock package working properly on windows 10. Not sure why, suspect interference from other python-related packages/paths.
>
> https://www.python.com
>
> after a large number of hours spent in python-install heck, i finally have anaconda package with python v3.1 mostly working.
>
> https://www.anaconda.com
>
> their installer fails if there is any extraneous python anywhere in path - got past that by uninstalling, manual removals, and manual editing of environment variables to get everything back to bare metal. another major time-waste was that anaconda also appears to succeed but actually fails if installed as localuser (somewhat similar failure to install of stock python) and so have installed it as admin, with permissions for all users, for best supervisory ability.
>
> launching the anaconda navigator and anaconda spyder environments still fails for me, but otherwise things seem to be working. tracked down idle and exposed its batch file. of note is that anaconda includes R and also Jupyter notebook. these are all important target environments for X3D.
>
> will keep working, hoping to set up a meeting next week after long weekend. given that i've finally got something going, am ready to resume "playing catch up" to your PYJNIUS mappings for X3DJSAIL.
>
> as part of resynch, let's start out with documenting X3dToPython similar to how we proceeded with
>
> X3D JSON Encoding
>
> http://www.web3d.org/wiki/index.php/X3D_JSON_Encoding
>
> X3D to JSON Stylesheet Converter
>
> http://www.web3d.org/x3d/stylesheets/X3dToJson.html
>
> Might Tuesday 1400-1500 pacific (or later) work for you? Web3D Conference line, in case someone wants to join the journey with us.
>
> Looking forward to regaining progress together, again thanks for your patience.
>
> all the best, Don
>
> --
>
> Don Brutzman Naval Postgraduate School, Code USW/Br brutzman at nps.edu
>
> Watkins 270, MOVES Institute, Monterey CA 93943-5000 USA +1.831.656.2149
>
> X3D graphics, virtual worlds, navy robotics http://faculty.nps.edu/brutzman
>
all the best, Don
--
Don Brutzman Naval Postgraduate School, Code USW/Br brutzman at nps.edu
Watkins 270, MOVES Institute, Monterey CA 93943-5000 USA +1.831.656.2149
X3D graphics, virtual worlds, navy robotics http://faculty.nps.edu/brutzman
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: material.future.py
URL: <http://web3d.org/pipermail/x3d-public_web3d.org/attachments/20190317/7738ccb2/attachment-0002.ksh>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: material.py
URL: <http://web3d.org/pipermail/x3d-public_web3d.org/attachments/20190317/7738ccb2/attachment-0003.ksh>
More information about the x3d-public
mailing list