编程知识 cdmana.com

Look at the second dimension! Wumeng music game score finder based on serverless

Preface

 The author of this article : Yuange made 

One 、 What is? Serverless Framework

Serverless Framework It is a very popular serverless application framework in the industry , Developers do not need to care about the underlying resources to deploy the complete available Serverless Application Architecture .Serverless Framework With resource choreography 、 Automatic telescopic 、 Event driven capabilities , Covering code 、 debugging 、 test 、 Deployment and other life cycles , Help developers through linkage cloud resources , Build quickly Serverless application

you 're right , Like I saw a few days ago 《Serverless Song of 》 It's said in it I'm gonna reduce your ops, It can greatly reduce the pressure of operation and maintenance , Let's get started ! Note that the development environment needs Node.js 10.0+, One click global installation :npm install -g serverless

Two 、 Tencent cloud Flask Serverless Component brief introduction

Tencent cloud Flask Serverless Component, Support Restful API Deployment of services

It's customary to deploy first demo Well

  1. Local PyCharm Create a new Flask project

Flask

  1. Manually create the content as Flask Of requirements.txt

  2. according to The configuration document establish serverless.yml, For example, the complete content of the actual use of this project , The first use can be simplified at your discretion

  3. Write the key to .env( Of course , When deploying, you can also select wechat code scanning authorization )

TENCENT_SECRET_ID=<rm>
TENCENT_SECRET_KEY=<rm>

 Successful deployment

 Successful visit

This is based on Serverless Of Flask Demo The deployment is complete , Next, continue to write the rest of the code in your own way .

3、 ... and 、maimai_DX

maimai It's an arcade video game .

Put a moving picture here and experience it by yourself , The original material comes from 「 External record maimai」QZKago Requiem Re:MASTER ALLPERFECT Player: Ruri*R

QZKago

At home , You can only view the results from the official account of WeChat , And every time you enter the page, you need to log in with wechat authorization , And the number of records stored in it is limited , Photo album Only the latest 10 strip , Game records Only the latest 50 strip ( It's a queue , First in, first out ). This is the original intention of this project , Every time you make a score, you should keep it .

Wumeng scoring device

The results show that , front end Fomantic-UI, Back end Flask+MySQL.gh Open source address :https://github.com/yuangezhizao/maimai_DX_CN_probe, welcome watchstarfork & pr

https://maimai.yuangezhizao.cn

At present, the following functions are installed :

  1. wechat_archive Contained in the Home page , Game data , Photo album and Game records : The original web page has been modified , And added Highcharts Library visualization curves show changes
  2. record contain Record ( Pagination ) and differences ( Pagination ): Self written quick preview page , It is a very practical function to view historical records and changes in performance
  3. info contain Shop list : That is, the basic information of all pavements , Output to a page , Easy to search within the page

The development process

Next, in chronological order , Describe the problems encountered in the development process and how to solve

1. Serverless Framework Component The configuration file

Serverless Framework Now it is V2 edition , That is to say, we can't follow the previous version of serverless.yml The configuration file , The document needs to be revised again .

a. Previous versions will be based on requirements.txt Automatically download the third party library to the project directory .serverless Under folder requirements Folder to participate in the final dependency packaging , Compressed into zip The file is finally uploaded to the cloud function running environment

b. The latest version is no longer automatically downloaded , It needs to be handled by itself . Reference usage of official examples :hook

  src:
    # TODO:  install python The project depends on the current directory of the project 
    hook: 'pip3 install -r requirements.txt -t ./requirements'
    dist: ./
    include:
      - source: ./requirements
        prefix: ../ # prefix, can make ./requirements files/dir to ./
    exclude:
      - .env
      - 'requirements/**'

The notes are very clear , Use hook According to requirements.txt Download the third party library to the project directory requirements Folder , Avoid third party libraries causing confusion in local folder management . then include In the project directory requirements The folder is in the cloud prefix, That is, for the cloud function running environment in the cloud ,requirements The third-party library and project directory in the folder are the same level , It can be imported and used normally . Yes, of course , The local operation uses a global third-party library , It is not used in the project directory requirements Folder .

2. Layer Management Overview

The former ( finger b) It's a very reasonable design , But in the actual environment, new problems have been found . Exactly the same configuration file

  src:
    hook: 'pip3 install -r ./src/requirements.txt -t ./src/requirements'
    dist: ./src
    include:
      - source: ./requirements
        prefix: ../
    exclude:
      - .env

stay macOS After successful deployment , In the cloud function editor, you can see requirements Folder does not exist , Third party libraries and project directories are peer to peer , No problem .

But in the Windows After successful deployment , In the cloud function editor, you can see requirements Folder ? That is, the third party library and the project directory are not the same level , So the interview will appear no module found The import error of ……

Try again and again to modify prefix The debugging of the configuration item is not successful in the end , Therefore, two solutions are proposed here :

a. Modify the configuration file as follows , Make the local third-party library and project directory exist at the same level

  src:
    hook: 'pip3 install -r ./src/requirements.txt -t ./src'
    dist: ./src
    exclude:
      - .env

But as projects and third-party libraries expand folders, more and more , It's very difficult to manage

b. Use the cloud function to provide layer

although sls deploy Deployment is fast , But it would be better to upload project code only when deployed without handling dependencies , So cross end collaboration development just needs to care about project code ok 了 , There's no need to manage dependencies anymore !

And a little bit more , Want to be in SCF Editing function code online in the console requires that the deployment package be kept in 10MB following , Don't think ten trillion is big , It's also possible to run out quickly

 Show only entry files

How to operate it ? That is to package and create third-party library folders directly as layers , In the function code, you can directly pass import quote , After all, there are special libraries like Brotli,Windows Under no vc++ You can only go to https://lfd.uci.edu/~gohlke/pythonlibs download wheel install .

macOS After normal installation, you will get _brotli.cpython-39-darwin.so,brotli.py And then with import _brotli The form of import , But there's a new problem , Errors will be reported in the cloud ModuleNotFoundError: No module named '_brotli'"

At present SCF The execution environment of is based on the following : standard CentOS 7.2

In order to solve the problem, try in linux Packaging in the environment , Pick up what you have CentOS 8.2 The virtual machine starts to operate

pip3 install -r requirements.txt -t ./layer --upgrade
zip -r layer.zip ./layer

And then you can pack it up layer.zip Download it locally and upload it again , For the time being, it can be done once and for all .

 Layer details

by the way , Configuration files can be removed hook And add layers

  src:
    src: ./src
    exclude:
      - .env
      - '__pycache__/**'
  layers:
    - name: maimai_DX_CN_probe
      version: 3

The bound layer function is triggered to run , When starting a concurrent instance , The running code of the loading function will be decompressed to /var/user/ Under the table of contents , At the same time, the contents of the layer will be decompressed and loaded to /opt Under the table of contents . If you need to use or access files file, Place in the root directory of the compressed file when creating the layer . After decompressing and loading , You can go directly through the catalog /opt/file Access to the file . If you create a layer , Compress by folder dir/file, When the function is running, it needs to pass /opt/dir/file Access to specific files

Experience faster deployment ! Because the third-party library is already packaged in “ layer ” It's in

But the strange thing is , Importing any third-party library in the cloud will result in an error , So I debugged and looked at path

for path in sys.path:
    print(path)

/var/runtime/python3
/var/user
/opt
/var/lang/python3/lib/python36.zip
/var/lang/python3/lib/python3.6
/var/lang/python3/lib/python3.6/lib-dynload
/var/lang/python3/lib/python3.6/site-packages
/var/lang/python3/lib/python3.6/site-packages/pip-18.0-py3.6.egg

Check it again opt

import os
dirs = os.listdir('/opt')

for file in dirs:
   print(file)

layer

That's what I realized , You need to package directly in the current path . After uploading “ layer ” Update to version 2, however ModuleNotFoundError: No module named '_brotli' The error is still reported , And confirm _brotli.cpython-38-x86_64-linux-gnu.so The file actually exists .

And in the CentOS and macOS There is no problem with local import , That's a big problem , I think it's probably python Version problem , So I went to look for ready-made 3.6 Environment , Here, for example. :

3.6.8

After uploading again “ layer ” Update to version 3, Successful visit ! The subject is finally solved , It turns out that it is necessary to Same version Of Python 3.6 Running environment

3. Custom entry file

components Source code tencent-flask/src/_shims/ The files in are repackaged and uploaded to the cloud function without any change , There are currently two documents

a. severless_wsgi.py, Role is converts an AWS API Gateway proxied request to a WSGI request. WSGI The full name is Python Web Server Gateway Interface namely Web Server gateway interface , It is for Python Defined by language Web The server and Web A simple and general interface between applications or frameworks

b. sl_handler.py, It's the default entry file

import app  # Replace with your actual application
import severless_wsgi

# If you need to send additional content types as text, add then directly
# to the whitelist:
#
# serverless_wsgi.TEXT_MIME_TYPES.append("application/custom+json")

def handler(event, context):
    return severless_wsgi.handle_request(app.app, event, context)

For your own project , Used Flask Of Factory function , To avoid being re modified in the cloud Function Editor every time , The best way is to customize the entry file :

import severless_wsgi

from maimai_DX_CN_probe import create_app  # Replace with your actual application


# If you need to send additional content types as text, add then directly
# to the whitelist:
#
# serverless_wsgi.TEXT_MIME_TYPES.append("application/custom+json")

def handler(event, context):
    return severless_wsgi.handle_request(create_app(), event, context)

Reassign Execution method by serverless_handler.handler, Just ok 了

4. url_for Output http Instead of https Of URL

Redirect to in view function url_for The generated links are http, instead of https…… Actually the question is Flask Documents Standalone WSGI Containers There is a description of

At the end of the day, it's not Flask The problem of , It is WSGI Environmental problems , The recommended method is to use middleware , The official also gave ProxyFix

from werkzeug.middleware.proxy_fix import ProxyFix
app.wsgi_app = ProxyFix(app.wsgi_app, x_proto=1, x_host=1)

But from X-Forwarded-Proto Take the value of ,apigw Among them is http, So you can't use this directly ProxyFix because Flask Our community is still perfect , Many predecessors have paved the way , So go straight Stack Overflow Search for solutions ,Flask url_for generating http URL instead of https The cause of the problem is shown in the figure :Browser ----- HTTPS ----> Reverse proxy(apigw) ----- HTTP ----> Flask Because I'm in apigw Set up Front end type only https, in other words Browser The end is impossible to use http Visited , By printing environ You know

{
  "CONTENT_LENGTH": "0",
  "CONTENT_TYPE": "",
  "PATH_INFO": "/",
  "QUERY_STRING": "",
  "REMOTE_ADDR": "",
  "REMOTE_USER": "",
  "REQUEST_METHOD": "GET",
  "SCRIPT_NAME": "",
  "SERVER_NAME": "maimai.yuangezhizao.cn",
  "SERVER_PORT": "80",
  "SERVER_PROTOCOL": "HTTP/1.1",
  "wsgi.errors": <__main__.CustomIO object at 0x7feda2224630>,
  "wsgi.input": <_io.BytesIO object at 0x7fed97093410>,
  "wsgi.multiprocess": False,
  "wsgi.multithread": False,
  "wsgi.run_once": False,
  "wsgi.url_scheme": "http",
  "wsgi.version": (1, 0),
  "serverless.authorizer": None,
  "serverless.event": "<rm>",
  "serverless.context": "<rm>",
  "API_GATEWAY_AUTHORIZER": None,
  "event": "<rm>",
  "context": "<rm>",
  "HTTP_ACCEPT": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
  "HTTP_ACCEPT_ENCODING": "gzip, deflate, br",
  "HTTP_ACCEPT_LANGUAGE": "zh-CN,zh;q=0.9,en;q=0.8",
  "HTTP_CONNECTION": "keep-alive",
  "HTTP_COOKIE": "<rm>",
  "HTTP_ENDPOINT_TIMEOUT": "15",
  "HTTP_HOST": "maimai.yuangezhizao.cn",
  "HTTP_SEC_FETCH_DEST": "document",
  "HTTP_SEC_FETCH_MODE": "navigate",
  "HTTP_SEC_FETCH_SITE": "none",
  "HTTP_SEC_FETCH_USER": "?1",
  "HTTP_UPGRADE_INSECURE_REQUESTS": "1",
  "HTTP_USER_AGENT": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.111 Safari/537.36",
  "HTTP_X_ANONYMOUS_CONSUMER": "true",
  "HTTP_X_API_REQUESTID": "5bcb29af2ca18c1e6d7b1ec5ff7b5427",
  "HTTP_X_API_SCHEME": "https",
  "HTTP_X_B3_TRACEID": "5bcb29af2ca18c1e6d7b1ec5ff7b5427",
  "HTTP_X_QUALIFIER": "$LATEST"
}

HTTP_X_FORWARDED_PROTO Corresponding apigw The variable in is HTTP_X_API_SCHEME, So the solution is as follows :app.wsgi_app = ReverseProxied(app.wsgi_app)

class ReverseProxied(object):
    def __init__(self, app):
        self.app = app

    def __call__(self, environ, start_response):
        scheme = environ.get('HTTP_X_FORWARDED_PROTO')
        if scheme:
            environ['wsgi.url_scheme'] = scheme
        return self.app(environ, start_response)

app = Flask(__name__)
app.wsgi_app = ReverseProxied(app.wsgi_app)

5. Response data compression

Whether it's IISApache still Nginx, All of them have compression function . After all, the external network uplink of the virtual machine you are using is only 1M bandwidth , After compression, the effect of shortening the first screen time is greatly improved . about Serverless, The response data is through API Gateway Transfer to client , So compression should also be its ability ( Although the Internet speed has been greatly improved , But the compression still has to be compressed ), However, it did not find …… See some js The framework provides compression capabilities , So I decided to add Flask Self compression function . simply , By subscribing @app.after_request Signal and call a third-party library brotli Of compress The method can ( Go before you write gh See if there are ready-made wheel extensions , Sure enough …… At first, I used Flask-Zipper, Later it was replaced by Flask-Compress Solved the problem Actually measured 3.1 MB The data used in this study is brotli The compression algorithm is reduced to 76.1 kB

 Before compression

 After the compression

br

6. apigw The impact of different paths in three environments

The default mapping is as follows :

ID Environment name Access path
1 Release release
2 pre-release prepub
3 test test

Because of the configuration static_url_path by "", namely static The folder is mapped to / The next path , So add releaseprepub and test It's natural to visit 404 了 So it's bound to Custom domain name , Use custom path mapping , And will Release The access path of the environment is set to /, So visit again Release There's no problem with the environment

ID Environment name Access path
1 Release /
2 pre-release prepub
3 test test

7. Simultaneous access Private networks and Extranet

Cloud functions There are several cloud databases available in

  • Cloud database CDB, need Private networks visit , Although can access through the external network, but can go to the intranet, do not go outside the network
  • PostgreSQL for Serverless(ServerlessDB), This is from the government Serverless Matched pg database
  • Development of cloud TCB Medium MongoDB, If you remember correctly, you need to open the internal test permission access

Because i migrated from the old website , The data has not been migrated yet , So directly access the original cloud database CDB, stay Cloud functions To configure Belong to the network and The subnet it belongs to that will do . But you can't access the Internet at this time , One solution is to turn on Public network access and The public network is fixed IP, You can access both intranet and extranet resources at the same time . About profiles , This project is Single instance application in other words Only one component is introduced into the project , Only one component instance is generated during deployment . But if you want to introduce a database , You have to add new components , Currently in Flask Components Database related configuration items are not provided in , Therefore need Multiple components are introduced into the project , Generate multiple component instances at deployment time . It's also very simple. , Create a containing serverless.yml New folder for , Used to configure postgresql

component: postgresql # ( Required )  Component name , Here is  postgresql
name: maimai_DX_CN_probe # ( Mandatory )  Component instance name .
org: yuangezhizao # ( Optional )  Used to record organizational information , The default value is your Tencent cloud account  appid, Must be a string 
app: yuangezhizao # ( Optional )  Used to record organizational information .  Default and name identical , Must be a string 
stage: dev # ( Optional )  Used to distinguish environmental information , The default value is  dev

inputs:
  region: ap-beijing #  Optional  ap-guangzhou, ap-shanghai, ap-beijing
  zone: ap-beijing-3 #  Optional  ap-guangzhou-2, ap-shanghai-2, ap-beijing-3
  dBInstanceName: maimai_DX_CN_probe
  #  projectId: 0
  dBVersion: 10.4
  dBCharset: UTF8
  vpcConfig:
    vpcId: vpc-mrg5ak88
    subnetId: subnet-hqwa51dh
  extranetAccess: false

And then at the terminal cd Go to this directory and execute sls deploy You can successfully deploy postgresql

yum install python3-devel postgresql-devel
pip install psycopg2

result

import psycopg2
File "/opt/psycopg2/__init__.py", line 51, in &lt;module&gt;
from psycopg2._psycopg import (                     # noqa
ImportError: libpython3.6m.so.1.0: cannot open shared object file: No such file or directory

The following problems are being solved :

  • http Force a jump https
  • Push test environment to production environment

thus , This is the end of the article , welcome communication

One More Thing

Experience Tencent cloud immediately Serverless Demo, receive Serverless New user gift pack serverless/start

Welcome to visit :Serverless Chinese net

版权声明
本文为[Tencent cloud serverless]所创,转载请带上原文链接,感谢

Scroll to Top