Tag Archives: python

Notepad

Issue installing pyinstaller

Leave a comment

Recently while running:

pip install pyinstaller

I encountered the following error:

Failed to build pyinstaller
Skipping wheel build for altgraph, due to binaries being disabled for it.
Skipping wheel build for pefile, due to binaries being disabled for it.
ERROR: Could not build wheels for pyinstaller which use PEP 517 and cannot be installed directly

It turns out that Windows Defender (the built in AV in Windows 10) was blocking runw.exe

The real error was a few lines above:

error: could not open 'PyInstaller\bootloader\Windows-32bit\runw.exe': Invalid argument

A quick unblock from Windows Defender and it worked.

Thanks to: https://github.com/pyinstaller/pyinstaller/issues/3813#issuecomment-575319940 for the hint.

Notepad

Upgrading requests_toolbelt in Zato 2.0.8

Leave a comment

After failing with pip and easy_install I came across this post again:

https://forum.zato.io/t/what-is-the-canonical-way-to-add-extra-python-packages-to-my-zato-server/580?source_topic_id=586

There is another way, that of updating versions.cfg and buildout.cfg files but if you are more familiar with pip than buildout then pip install package-name is perfectly fine.

So it turns out it is as easy as:

nano ~/current/versions.cfg
update the version from 0.2.0 to 0.8.0 then run
cd ~/current
buildout

And buildout will do it’s magic.

Notepad

Zato: Add MSSQL to Outgoing SQL Options

Leave a comment

To Connect with MSSQL or SQL Server via Zato 3:

#sudo su - zato
$/opt/zato/current/bin/pip install pymssql
(More info: http://pymssql.org/en/stable/intro.html)
$nano /opt/zato/env/qs-1/server1/config/repo/sql.conf

Add the following lines at the end:
[mssql+pymssql]
display_name=MSSQL
sqlalchemy_driver=mssql+pymssql
ping_query=SELECT 1

???I don’t know if this is required or not, but I still do it.
Deploy to other servers in the cluster:
$cp /opt/zato/env/qs-1/server1/config/repo/sql.conf /opt/zato/env/qs-1/server2/config/repo/

Restart your cluster and MSSQL should appear as an option under SQL Outgoing connections.

Clippings

Zato MySQL Connection with UTF-8

Leave a comment

Today this error appeared in an Zato Service:

UnicodeEncodeError: 'latin-1' codec can't encode character u'\u2019' in position 1: ordinal not in range(256)

I turns out that SQLAlchemy defaults to latin-1 encoding instead of UTF-8 for MySQL connections.

Thanks to Andrea on a Zato Thread here: https://mailman-mail5.webfaction.com/pipermail/zato-discuss/2015-November/001443.html points out that we can just add
dbname?charset=utf8 in the WebUI.

While not intuitive, it does work.
I added it, had the service try the action again and all was well in the world again.

Clippings

SuiteCRM / SugarCRM Rest API 500 Error get_entry_list

Leave a comment

I stumbled upon this problem today when trying out the SuiteCRM Rest API.
When searching for an account by it’s name I was using a search like this:

"name = 'Andrew'"

The API was returning a 500 error.
I dug into the suitecrm.log and found this:

where (name = 'Andrew') AND accounts.deleted=0: MySQL error 1052: Column 'name' in where clause is ambiguous

It seems that because of the joins, it can’t work out which ‘name’ we are talking about. Lets add the table name in and tada it works:

"accounts.name = 'Andrew'"

Clippings

Zato and Pandas (strange combination I know)

Leave a comment

I’ve been working with Zato for a project and I’m getting some random text out of a HTTP request that looks like a CSV file, so the easiest way I know how to deal with this CSV data is with Pandas, then I could process it enough and clean it up to hand back to Zato.

But Zato doesn’t ship with Pandas….
(TL;DR at the bottom of the post)

In theory this should work:

/opt/zato/2.0.7/bin/pip install pandas

But then I get the error when I do the following:

/opt/zato/2.0.7/bin/python
>>> import pandas as pd
File "/opt/zato/current/local/lib/python2.7/site-packages/pandas/__init__.py", line 19, in
"Missing required dependencies {0}".format(missing_dependencies))
ImportError: Missing required dependencies ['pytz', 'dateutil']

No luck.

After trying pip install pytz and pip install –upgrade pytz
I get this error:

Installing collected packages: pytz
Found existing installation: pytz 2014.4
Cannot remove entries from nonexistent file /opt/zato/2.0.7/eggs/easy-install.pth

But this does work:

/opt/zato/2.0.7/bin/easy_install --upgrade pytz

Now to fix dateutil.

easy_install --upgrade python-dateutil

Let’s try it again:

/opt/zato/2.0.7/bin/python
>>> import pandas as pd
Traceback (most recent call last):
File "", line 1, in
File "/opt/zato/current/local/lib/python2.7/site-packages/pandas/__init__.py", line 23, in
from pandas.compat.numpy import *
File "/opt/zato/current/local/lib/python2.7/site-packages/pandas/compat/__init__.py", line 361, in
from dateutil import parser as _date_parser
File "/opt/zato/current/local/lib/python2.7/site-packages/python_dateutil-2.6.0-py2.7.egg/dateutil/parser.py", line 40, in
from six import text_type, binary_type, integer_types
ImportError: No module named six

Bummer!


easy_install six

And for a final test:

/opt/zato/2.0.7/bin/python
>>> import pandas as pd
>>>

Yay, it’s working. Hopefully, nothing in Zato has broken…

TL;DR:


/opt/zato/2.0.7/bin/easy_install pandas
/opt/zato/2.0.7/bin/easy_install --upgrade pytz
/opt/zato/2.0.7/bin/easy_install --upgrade pytz
/opt/zato/2.0.7/bin/easy_install six

UPDATE:
(I have not tested this)
It might work better to use buildout to make this work see this comment here:

https://forum.zato.io/t/what-is-the-canonical-way-to-add-extra-python-packages-to-my-zato-server/580?source_topic_id=586

There is another way, that of updating versions.cfg and buildout.cfg files but if you are more familiar with pip than buildout then pip install package-name is perfectly fine.

However, if a dependency is not on PyPI nor any other package index (such as your own internal one) then you can place individual Python modules in zato_extra_paths too.

Notepad

Selenium / Python Notes June 2016

Leave a comment

For the past 24 hours I’ve been jumping into some Selenium fun in Python.

Notes:

  • Firefox 47 doesn’t work, however Firefox 47.0.1 does.
Selenium Setup


from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.proxy import *
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException

def init_driver():
driver = webdriver.Firefox()
driver.wait = WebDriverWait(driver, 5)
return driver

def lookup(driver):
driver.get(“http://www.google.com”)

if __name__ == “__main__”:
driver = init_driver()
lookup(driver)
time.sleep(5)
driver.quit()

Two ways to interact with an element on the page:


try:
textbox = driver.wait.until(EC.presence_of_element_located((By.ID, "formTxtBox")))
except TimeoutException:
print("Login Form Not Found!")

or

textbox = driver.find_element_by_id("formTxtBox")

Taking HTML and feeding it into BeautifulSoup


div = driver.find_element_by_id("id_of_div_here")
soup = BeautifulSoup(propertyDropDown.get_attribute('innerHTML'), 'html.parser')
#Do something with the soup....

Creating a list of dates (requires pandas)


import pandas as pd
dates = []
index = pd.date_range('2015-7-1', periods=52, freq='W-WED')
for i in index:
dates.append(i.to_datetime())
return dates

Convert Table into CSV with BeauitfulSoup

(there are probably 100 better ways to do this…)

#Get the element of the table using CSS Selector
table = driver.wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, "#table > tbody")))
#Take our table and pass it into BeautifulSoup so we can easily transverse it.
soup = BeautifulSoup(table.get_attribute('innerHTML'), 'html.parser')
#Make an empty list
rows = []
#Loop over the soup
for row in soup.find_all('tr'):
rows.append([val.text.strip('\n').encode('utf8').strip('\xc2\xa0') for val in row.find_all('td')])
 
#Save out to CSV.
with open('output_file.csv', 'ab') as f:
writer = csv.writer(f)
writer.writerows(row for row in rows if row)

Disclaimer: I’ve been programming in Python for about 5 minutes and using Selenium for about 2….while this code worked for my project, it’s most likely littered with errors and bad practices. This is mostly just a reference for myself for a project I might need to reefer back to this stuff in 6-12 months time. I am definitely not the person to ask for help from when it comes to this stuff.

Clippings

Xero API pyXero and Anaconda Python

Leave a comment

Personal TL;DR notes on getting this to work, not a tutorial.

Download Install Anaconda
– At the time of writing it didn’t create Windows Shortcuts, fix this with openning cmd as admin

‘c:\Anaconda2\scripts\conda.exe install menuinst’

Agree to Updates

‘c:\Anaconda2\scripts\conda.exe install -f console_shortcut ipython ipython-notebook ipython-qtconsole launcher spyder’

Agree to updates.

Now there should be shortcuts in:

C:\ProgramData\Microsoft\Windows\Start Menu\Programs\Anaconda2 (64-bit)

Now onto Xero

  • git clone https://github.com/freakboy3742/pyxero.git
  • Open up Anaconda Prompt (Anaconda2x64)
  • cd into folder with git of pyxero
  • use command: python setup.py install
  • pip uninstall oauthlib
  • pip install oauthlib==0.7.2

The last two commands fix the error: ImportError: No module named jwt.algorithms

Again these are personal notes, not a full blown tutorial.