Scrapy學習筆記

若要整合Django和Scrapy，可利用DjangoItem，可節省定義Item的部分，讓Scrapy的Item定義自動去參照Django model，也可在Item增加欄位。

ItemLoader的部分，預設add_value後，postprocessor都會以list為最後輸出結果，可更感post processor為TakeFirst。

在Pipeline裡，若應用到DjangoItem，要先save後，在以Django model來操作 Foreign Key或是ManyToManyField部分。DjangoItem無法直接取得Django model內容。DjangoItem只是取去參考Django model，省去寫 field_xxx = scrapy.Field()而已。

pandas Dataframe操作

import pandas as pd
df = pd.read_excel('b.xlsx', 'Acceptance')
df[df['Config 3'].notna()]

裡面的df['Config 3'].notna()是filter的方式，結果是列出所有Config13欄位不是na的 rows

Pandas的Series是縱向的，例如某個Column的值的列表

pandas兩個主要的data structure
DataFrame和Series
DataFrame就是Series的容器
Series就是純量的容器

Indexing / Selection

The basics of indexing are as follows:

Operation	Syntax	Result
Select column	df[col]	Series
Select row by label	df.loc[label]	Series
Select row by integer location	df.iloc[loc]	Series
Slice rows	df[5:10]	DataFrame
Select rows by boolean vector	df[bool_vec]	DataFrame

如果要一個row一個row去做動作

https://stackoverflow.com/questions/16476924/how-to-iterate-over-rows-in-a-dataframe-in-pandas

df.iterrows()

inotify-tools監測資料夾

https://www.howtogeek.com/405468/how-to-perform-a-task-when-a-new-file-is-added-to-a-directory-in-linux/amp/

Delopy Django project on Ubuntu 1804 with mod_wsgi

1. Install apache2 & postgresql
2. Install mod_wsgi
# sudo apt install libapache2-mod-wsgi-py3
3. Create a Python3 virtualenv
# mkvirtualenv -p $(which python3) django & psycopg2-binary
(django)# pip install django
(django)# pip install psycopg2-binary
4. Enter into project folder. /home/jack/Documents/Projects/myproject
5. Install Requirement python package
(django) # pip install -r requirements.txt
6. Note that the wsgi.py is put on /home/jack/Documents/Projects/myproject/myproject
7. Edit settings.py in ./myproject
Add ALLOWED_HOSTS = ['*',]
(This is risky)
Add STATIC_ROOT = '/home/jack/Documents/Projects/myproject/static'
8. Setup Postgresql database correctly & create a database in it.
9. Edit settings.py to meet the database you want to use.
10. Database Migrate & collectstatic
(django) # python manage.py migrate
(django) # python manage.py collectstatic
11. create a new conf in /etc/apache2/site-available myproject.conf with sudo

    ServerName myproject
    Alias /robots.txt /home/jack/Documents/Projects/myproject/static/static/robots.txt
    Alias /favicon.ico /home/jack/Documents/Projects/myproject/static/favicon.ico
    Alias /static/ /home/jack/Documents/Projects/myproject/static/
   
        Require all granted
   
    WSGIDaemonProcess django python-home=/home/jack/.virtualenvs/django/ python-path=/home/jack/Documents/Projects/myproject/
    WSGIScriptAlias / /home/jack/Documents/Projects/myproject/myproject/wsgi.py
    WSGIProcessGroup django
   
       
        Require all granted
       
   

12. Edit /etc/apache2/ports.conf, Add a line

Listen 4002

13. Enable the site & reload

# sudo a2ensite myproject

# sudo systemctl reload apache2

14. Allow port 4002 in Firewall

# sudo ufw allow 4002

Reference:
https://docs.djangoproject.com/en/2.1/howto/deployment/wsgi/
https://docs.djangoproject.com/en/2.1/howto/deployment/checklist/

TestLink setup on CentOS 7

#Install PostgreSQL: https://www.digitalocean.com/community/tutorials/how-to-install-and-use-postgresql-on-centos-7
# sudo yum install postgresql-server postgresql-contrib
# sudo postgresql-setup initdb
# sudo systemctl start postgresql
# sudo systemctl enable postgresql

https://dotblogs.com.tw/jovepaterlab/2017/04/17/222958
[root@localhost peter]# su - postgres
-bash-4.2$ psql -U postgres
psql (9.6.2)
Type "help" for help.

postgres=# ALTER USER postgres WITH PASSWORD 'password'
postgres-# \q
-bash-4.2$ exit
logout

Edit /var/lib/pgsql/data/pg_hba.conf
local    all             all                                    md5
host    all             all             127.0.0.1/32            md5
host    all             all             ::1/128                 md5

https://www.tecmint.com/install-php-7-in-centos-7/
# yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
# yum install http://rpms.remirepo.net/enterprise/remi-release-7.rpm
# yum-config-manager --enable remi-php72
# yum install php php-mcrypt php-cli php-gd php-curl php-pgsql php-ldap php-zip php-fileinfo
# php -v

Enable Apache2
# systemctl start httpd
# systemctl enable httpd

Edit /etc/php.ini
session.gc_maxlifetime = 2400
max_input_vars = 3000
memory_limit = 256M
upload_max_filesize = 20M
max_execution_time = 120


Set MySQL password
mysqladmin -u root password password

# Copy Testlink source code
# wget -e use_proxy=yes -e https_proxy=jack_lin3.compal.com:3128 https://github.com/TestLinkOpenSourceTRMS/testlink-code/archive/testlink_1_9.zip
   47  unzip
# unzip testlink_1_9.zip -d /var/www/html/
# cd /var/www/html/
# ls
# mv testlink-code-testlink_1_9/ testlink/
# ls
# chown -R apache:apache testlink
mkdir  /var/testlink
mkdir /var/testlink/logs
mkdir /var/testlink/upload_area
chown apache:apache /var/testlink/logs
chown apache:apache /var/testlink/upload_area


Check firewall setting
https://linuxhint.com/list_open_ports_firewalld/
https://www.digitalocean.com/community/tutorials/how-to-set-up-a-firewall-using-firewalld-on-centos-7

Add http service in firewall
# sudo firewall-cmd --zone=public --permanent --add-service=http

link1
link2
# sudo chcon -Rv --type httpd_sys_rw_content_t /var/www/html/testlink/gui/templates_c
##### chcon -Rv --type httpd_sys_rw_content_t /var/www/html/testlink
https://tripal.info/tutorials/v3.x/installation/server-setup/centos7
# setsebool -P httpd_can_network_connect_db on

Remove line 872~879
/install/sql/postgres/testlink_create_tables.sql
https://github.com/TestLinkOpenSourceTRMS/testlink-code/blob/testlink_1_9/install/sql/postgres/testlink_create_tables.sql


Open http:///testlink, Follow steps to install.

YOU NEED TO RUN MANUALLY Following Script on your DB CLIENT Application

/var/www/html/testlink/install/sql/postgres/testlink_create_udf0.sql


YOUR ATTENTION PLEASE:
To have a fully functional installation You need to configure mail server settings, following this steps

    copy from config.inc.php, [SMTP] Section into custom_config.inc.php.
    complete correct data regarding email addresses and mail server.


Installation was successful!
You can now log in to Testlink (using login name:admin / password:admin - Please Click Me!).

Deploying Flask App with mod_wsgi (Python3) in Ubuntu 1804

1. Install apache2
2. Install mod_wsgi
# sudo apt install libapache2-mod-wsgi-py3
3. Create a Python3 virtualenv with Flask installed
# mkvirtualenv -p $(which python3) flask
(flask)# pip install flask
4. Create a .wsgi file for project (put in project folder)
newsite.wsgi:

activate_this = '/home/jack/.virtualenvs/flask/bin/activate_this.py'
with open(activate_this) as file_:
    exec(file_.read(), dict(__file__=activate_this))
import sys
sys.path.insert(0, '/home/jack/Documents/Projects/newsite')
from app import app as application

5. Open Apache2 ports. Edit file /etc/apache2/ports.conf

Listen 4002

6. Create a file for new site in /etc/apache2/sites-available
newsite.conf:


    ServerName newsite

    WSGIDaemonProcess newsite user=jack group=jack threads=5
    WSGIScriptAlias / /home/jack/Documents/Projects/newsite/newsite.wsgi

   
        WSGIProcessGroup newsite
        WSGIApplicationGroup %{GLOBAL}
        Require all granted
   

7. Enable the newsite
# sudo a2ensite newsite
(a2dissite for disable)

8. Reload the Apache2
# sudo systemctl reload apache2

9. Allow port 4002 by ufw
# sudo ufw allow 4002
(sudo ufw delete allow 4002) to disable

10. Open broswer to browse http://:4002

Note1. Apache2 error log is in /var/log/apache2/error.log

Note2. If you are using open() in app.py, need to instead to app.open_resource() to prevent FileNotFoundException

Reference:
http://flask.pocoo.org/docs/1.0/deploying/mod_wsgi/
http://flask.pocoo.org/docs/0.12/api/#flask.Flask.open_resource

iSCSI target setup (Ubuntu)

https://www.server-world.info/en/note?os=Ubuntu_18.04&p=iscsi&f=1
https://wiki.archlinux.org/index.php/ISCSI/LIO

# create a directory

root@dlp:~# 

mkdir /var/lib/iscsi_disks 

# enter the admin console

root@dlp:~# 

targetcli 

targetcli shell version 2.1.fb43
Copyright 2011-2013 by Datera, Inc and others.
For help on commands, type 'help'.

/> cd backstores/fileio 

# create a disk-image with the name "disk01" on /var/lib/iscsi_disks/disk01.img with 10G
/backstores/fileio> create disk01 /var/lib/iscsi_disks/disk01.img 10G 
Created fileio disk01 with size 10737418240
/backstores/fileio> cd /iscsi 

# create a target
# naming rule : [ iqn.(year)-(month).(reverse of domain name):(any name you like) ]
/iscsi> create iqn.2018-05.world.srv:dlp.target01 
Created target iqn.2018-05.world.srv:dlp.target01.
Created TPG 1.
Global pref auto_add_default_portal=true
Created default portal listening on all IPs (0.0.0.0), port 3260.
/iscsi> cd iqn.2018-05.world.srv:dlp.target01/tpg1/luns 

# set LUN
/iscsi/iqn.20...t01/tpg1/luns> create /backstores/fileio/disk01 
Created LUN 0.
/iscsi/iqn.20...t01/tpg1/luns> cd ../acls 

# set ACL (it's the IQN of an initiator you permit to connect)
/iscsi/iqn.20...t01/tpg1/acls> create iqn.2018-05.world.srv:www.initiator01 
Created Node ACL for iqn.2018-05.world.srv:www.initiator01
Created mapped LUN 0.
/iscsi/iqn.20...t01/tpg1/acls> cd iqn.2018-05.world.srv:www.initiator01 

# set UserID for authentication
/iscsi/iqn.20...w.initiator01> set auth userid=username 
Parameter userid is now 'username'.
/iscsi/iqn.20...w.initiator01> set auth password=password 
Parameter password is now 'password'.
/iscsi/iqn.20...w.initiator01> exit 
Global pref auto_save_on_exit=true
Last 10 configs saved in /etc/rtslib-fb-target/backup.
Configuration saved to /etc/rtslib-fb-target/saveconfig.json

# after configuration above, the target enters in listening like follows

root@dlp:~# 

ss -napt | grep 3260 

LISTEN   0         256                 0.0.0.0:3260             0.0.0.0:* 

Disable Authentication

Navigate targetcli to your target (i.e. /iscsi/iqn.../tpg1) and:

.../tpg1> set attribute authentication=0