엔지니어 대한민국

Python Pickling

Posted on March 30, 2021February 9, 2023 by 솔천

Python에서 Pickling을 한다는건 어떤 Object를 serializing해서 어딘가에 저장하고, 그걸 다시 꺼내서 de-serializing하는 것을 pickling이라고 합니다. 어떤 오브젝트를 시리얼라이징함으로 인해서 파일에 저장할수 있게 하는거죠. 파이썬에 있는 어떤 Object든지 피클링을할수 있어요. 다시말해 피클링은 파이썬 오브젝트들 예를 들면 list, dict같은 애들을 문자열로 만들어 주는 작업을 말합니다. 이 작업은 문자열로 만들었던 정보가 다시 오브젝트로 재건되는데 필요한 모든 정보를 해당 문자열 안에 가지고 있어야합니다. 아래는 어떤 오브젝트를 피클링해서 저장하고, 다시 가져와서 오브젝트로 만들어 주는 스크립트입니다.

import pickle
  
def storeData():
    # initializing data to be stored in db
    Omkar = {'key' : 'Omkar', 'name' : 'Omkar Pathak',
    'age' : 21, 'pay' : 40000}
    Jagdish = {'key' : 'Jagdish', 'name' : 'Jagdish Pathak',
    'age' : 50, 'pay' : 50000}
  
    # database
    db = {}
    db['Omkar'] = Omkar
    db['Jagdish'] = Jagdish
      
    # Its important to use binary mode
    dbfile = open('examplePickle', 'ab')
      
    # source, destination
    # print(pickle.dumps(db))
    pickle.dump(db, dbfile)                     
    dbfile.close()
  
def loadData():
    # for reading also binary mode is important
    dbfile = open('examplePickle', 'rb')     
    # print(pickle.loads(dbfile))
    db = pickle.load(dbfile)
    for keys in db:
        print(keys, '=>', db[keys])
    dbfile.close()
  
if __name__ == '__main__':
    storeData()
    loadData()

위의 코드를 실행하면 아래와 같이 문자열로 저장되어있던 내용이 다시 오브젝트로 로딩이 되어서 보여집니다.

Omkar => {'age': 21,  'name': 'Omkar Pathak',  'key': 'Omkar',  'pay': 40000}
Jagdish => {'age': 50,  'name': 'Jagdish Pathak',  'key': 'Jagdish',  'pay': 50000}

Source: https://www.geeksforgeeks.org/understanding-python-pickling-example

Python Basic: Reverse Dict Mapping

Posted on March 16, 2021March 16, 2021 by 솔천

아래에 class라는 dict가 하나 정의되어 있습니다. 번호와 이름이 맵핑되어있는 dict에요.

my_class = { \
    "1": "ellie", \
    "2": "kevin", \
    "3": "daniel" \
}

여기서 번호를 가지고 이름을 찾을때는 my_class[번호]로 이름을 찾을수가 있는데 반대로 이름을 가지고 번호를 찾고 싶을때는 배열을 뒤집어야겠죠. 아래와 같이 tuple쌍을 배열로 만들고 dict로 변환할수도 있지만

new_dict = dict([(val, key) for key, val in my_class.items()])

바로 dict로 만들면 더 간단명료해지겠죠?

new_dict = {val: key for key, val in my_class.items()}

이렇게 key와 value를 뒤집은 배열을 만든뒤, new_dict[번호]를 반환해주면 되겠죠?

Find Denied Field in JSON

Posted on March 10, 2021March 10, 2021 by 솔천

When you take JSON data, you may want to reject certain fields from the JSON structure. Here’s an algorithm to compare the source JSON and the denied fields list and return True if there’s any field that is not allowed in the source JSON.

# Here's a function that checks if there's any not-allowed-fields are included in the source JSON data
def is_denied_field_in_json(deny={}, source={}):
    for key, t_val in deny.items():
        s_val = source.get(key)
        if s_val is None:
            continue
        elif not isinstance(t_val, dict):
            return True
        elif isinstance(t_val, dict)\
            and isinstance(s_val, dict)\
            and is_denied_field_in_json(t_val, s_val):
                return True
    return False

# This is the JSON data a user tries insert
source = {
    "a": 1,
    "b": {
        "bb": 2,
    },
    "c": 3,
    "d": {
        "dd": {
            "ddd": {
                "dddd": 4
            }
        }
    }
}

# Here're the test deny lists you can define in your code
deny_list_true1 = {
    "a": 0,
}
deny_list_true2 = {
    "n": 0,
    "a": 0,
}
deny_list_true3 = {
    "b": {
        "bb": 0
    }
}
deny_list_true4 = {
    "b": {
        "n": 0
    },
    "c": 3
}
deny_list_true5 = {
    "d": {
        "dd": {
            "ddd": 0
        }
    }
}
deny_list_false1 = {
    "n": 0,
}
deny_list_false2 = {
    "b": {
        "n": 0
    }
}
deny_list_false3 = {
    "d": {
        "n": {
            "dd": {
                "ddd": 0
            }
        }
    }
}
print(is_denied_field_in_json(deny_list_true1, source))
print(is_denied_field_in_json(deny_list_true2, source))
print(is_denied_field_in_json(deny_list_true3, source))
print(is_denied_field_in_json(deny_list_true4, source))
print(is_denied_field_in_json(deny_list_true5, source))
print(is_denied_field_in_json(deny_list_false1, source))
print(is_denied_field_in_json(deny_list_false2, source))
print(is_denied_field_in_json(deny_list_false3, source))

SQLAlchemy Tutorial

Posted on March 9, 2021March 9, 2021 by 솔천

Starting from the basic connection, this lecture will tell you how to get the total number of the grouped subquery records.

Basic connection

# Access the app container
docker exec -it container_id sh

# Run python
python

# Connect to the database
import sqlalchemy as db
engine = db.create_engine('mysql+pymysql://root:root@container_name/db_name')
connection = engine.connect()

# Connection Test
metadata = db.MetaData()
table_name = db.Table('table_name', metadata, autoload=True, autoload_with=engine)

# Print all columns in the table
table_columns = table_name.columns.keys()

Query

query = db.select([table_name]).limit(1)
result_proxy = connection.execute(query)
result_set = result_proxy.fetchall()

Session

from sqlalchemy.orm import sessionmaker

Session = sessionmaker(bind=engine)
session = Session()

Model

from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()

class TableName(Base):
    __tablename__ = 'table_name'
    id = db.Column('key', primary_key=True)
    name = db.Column('title')

class AnotherTableName(Base):
    __tablename__ = 'another_table_name'
    id = db.Column('id', primary_key=True)
    tablename_id = db.Column('tablename_id')
    title = db.Column('title')

result = session.query(db.func.count(TableName.id),TableName.id).group_by(TableName.id).all()

Join

query = session.query(TableName)
query = query.outerjoin(AnotherTableName, TableName.id == AnotherTableName.tablename_id)
query = query.filter(AnotherTableName.title == "apple")
result = query.all()

Declare the Query

mysql> select count(*) from table_name group by id

+----------+
| count(*) |
+----------+
|        1 |
...
|        1 |
|        1 |
|        1 |
+----------+
1483 rows in set (0.25 sec)

mysql> select count(*) from (select count(*) from table_name group by id) a;
+----------+
| count(*) |
+----------+
|     1483 |
+----------+
1 row in set (0.01 sec)

Group by

query = session.query(TableName.id, func.count('*').label("count"))
query = query.outerjoin(AnotherTableName, TableName.id == TableName.tablename_id)
query = query.filter(AnotherTableName.title == 'apple'))
query = query.group_by(TableName.id)
result = query.all()

Subquery

query = session.query(TableName.id, func.count('*').label("count"))
query = query.outerjoin(AnotherTableName, TableName.id == AnotherTableName.tablename_id)
query = query.filter(AnotherTableName.title == 'apple'))
query = query.group_by(TableName.id)
t = query.subquery()

total = session.query(func.count(t.c.count)).scalar()

References

Flask + uWSGI + Docker

Posted on February 24, 2021February 24, 2021 by 솔천

Flask

Install Flask.

$ pip install flask

Make an application.

$ vi hello.py
from flask import Flask
app = Flask(__name__)

@app.route("/")
def hello():
    return "Hello World!"

Test application.

$ FLASK_APP=hello.py flask run --port 5555

Check the result.

uWSGI

Create a uWSGI configuration file.

$ vi uwsgi.ini
[uwsgi]
module = hello:app
uid = www-data
gid = www-data
master = true
processes = 1

http = 0.0.0.0:5000
chmod-sock = 664
vacuum = true

die-on-term = true

Run uWSGI.

$ uwsgi --ini uwsgi.ini

Check the result.

Docker

Make a file to list the required modules to install in the docker container.

$ vi requirements.txt
Flask==1.0.2
uWSGI==2.0.17.1

Create Dockerfile.

FROM python:3.6-slim

COPY ./hello.py /app/hello.py
COPY ./uwsgi.ini /app/uwsgi.ini
COPY ./requirements.txt /app/requirements.txt

WORKDIR /app

RUN apt-get clean \
    && apt-get -y update

RUN apt-get -y install python3-dev \
    && apt-get -y install build-essential

RUN pip install -r requirements.txt --src /usr/local/src

CMD ["uwsgi", "--ini", "uwsgi.ini"]

Build Docker image

$ docker build . -t app_image

Run Docker

$ docker run --name app_container -p 80:5000 app_image

Check the result

References:

Simple uWSGI example with Python

Posted on February 24, 2021February 24, 2021 by 솔천

Install uWSGI

$ pip install uwsgi

Make an application

$ vi foobar.py
def application(env, start_response):
    start_response('200 OK', [('Content-Type','text/html')])
    return [b"Hello World"]

Run uWSGI

$ uwsgi --http :9090 --wsgi-file foobar.py

Check the result

References:

Quickstart for Python/WSGI applications

Datadog Agent

Posted on February 15, 2021February 15, 2021 by 솔천

여러분이 application에서 metrics를 Datadog agent에 보내면, agent는 매 15초마다 접수된 모든 metrics데이타를 취합해서 Datadog server로 한꺼번에 전송합니다.

Source: https://docs.datadoghq.com/agent/basic_agent_usage/?tab=agentv6v7

Static Inner Class in Python

Posted on January 19, 2021 by 솔천

class outer:
    class inner:
        @staticmethod
        def callme():
            print('you called me')

outer.inner.callme()

Manually Install Python Packages

Posted on January 11, 2021 by 솔천

만약 여러분의 local machine에 Python Package source code가 있다면, pip install을 이용하지 않고 모듈을 설치하는 방법이 있습니다. 바로 setup.py를 실행하는 방법인데요. 이 파일은 여러분이 Python Package를 만들고자 할때 root folder에 반드시 넣어야하는 파일인데요, python package를 만드는 방법에 대해서는 다음에 자세히 다루도록하겠습니다. 우선 소스코드가 있는 폴더로 이동을 하신후에 아래의 명령을 실행해주세요. 그러면 로컬 머신에 패키지가 설치가 됩니다.

python setup.py install

ES DSL Field Names

Posted on January 5, 2021 by 솔천

You can add a class method to your Document class.

    @classmethod
    def get_field_names(cls):
        """
        Return all ES field names
        """
        return cls._doc_type.mapping.properties.properties._d_.keys()